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Abstract 

This paper studies whether pupil performance gains achieved by autonomous schools — specifically 
academy schools in England — can be attributed to the strategic exclusion of poorly performing pupils. 
In England there have been two phases of academy school introduction, the first in the 2000s being a 
school improvement programme for schools serving disadvantaged pupil populations, the second a 
mass academisation programme in the 2010s which by contrast enabled better performing schools to 
become academies. Overall, on average across both programmes, exclusion rates are higher in 
academy schools. When the two programmes are considered separately, the earlier programme 
featured a much higher increase in the incidence of permanent exclusion. However, a number of 
simulated counterfactual experiments based on the statistical estimates show that rather than being 
used as a strategic means of manipulation to boost measured school performance, the higher rate of 
exclusion is instead a feature of the rigorously enforced discipline procedures that the pre-2010 
academies adopted. 
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1. Introduction 

Autonomous schools - like academy schools in England or charter schools in the US - are able to 
operate more independently than standard state or public schools. Depending on context, they tend 
to have more control over many operational issues like staffing, curriculum, budget setting and 
service provision, plus a range of additional freedoms that they can exercise in running their school. 
However, at the same time, in most settings the academic performance of their students is assessed 
in the same way as state/public schools. Therefore, when league tables or other school 
accountability measures based on student performance are constructed, the rankings of autonomous 
and non-autonomous schools are typically based on the same measures of performance. 

In England the headline figure by which secondary schools have been ranked in the 
annually published league tables (at least until recently) is the percentage of pupils getting five or 
more grade A* to C results in the final year of compulsory schooling examinations (taken in year 
11 at age 16). These high stakes exams are the General Certificate of Secondary Education (GCSE) 
exams taken at the end of secondary school. The school-level rankings based on pupils’ GCSE 
performance matter on a number of counts, including for enrolment of new pupils (as they are 
heavily scrutinised by the parents of potential new enrolees) and for evaluating head teacher and 
teacher performance (including salaries and potential pay for performance bonuses). They are 
widely available and highly accessible, for example being published in newspapers and on internet 
websites. 

There is now a quite sizable, and growing literature, about test score manipulation that may 
occur because of these kinds of incentivisation (see the recent review by Battistin, 2016). However, 
in the English educational environment, it is difficult to directly manipulate pupils' GCSE 


performance. Whilst part of the evaluation of the high stakes end of secondary school GCSE 


assessments are based on in year teacher assessment, the examinations that comprise the bulk of 
the scores are completely independent. They are marked externally through graders from a number 
of examination boards for different GCSE subjects.! 

One possibility that has received public attention about academy schools is that the greater 
autonomy with which these schools are able to operate has offered a means to manipulate their 
results in a different way, namely by differentially excluding poorly performing pupils from their 
on-roll lists. This could potentially explain some of the positive GCSE gains recorded by some of 
these academy schools (see Eyles and Machin, forthcoming), rather than there being a genuine 
improvement in the quality of education provided. 

This study empirically assesses this claim using a research design that derives causal 
estimates of the impact of academy enrolment on permanent exclusion in year 11, the final year of 
compulsory schooling in England. Permanent exclusion is a novel means of possible school results’ 
manipulation that has not been investigated in the literature to date.* The empirical analysis is 
undertaken for all secondary academies together and separately for the different waves of academy 
schools that have progressively appeared over time in the English secondary school landscape. 

The history of academisation of the English secondary school sector is outlined in more 
detail later in the paper (on this, see also Eyles aet al., 2016a, or Eyles et al., forthcoming), but this 
involves a separate comparison for three groups of academy schools that have opened over time: 
pre-2010 sponsored academies, post-2010 sponsored academies and post-2010 converter 
academies. The first of these were the pioneering academy schools set up under the Labour 
government of 1997-2010, which in their pre-academy life were almost all very disadvantaged 
schools and which, together with a new sponsor, became academy schools between 2003 and 2010. 


' See Machin, McNally and Ruiz-Valenzuela (2017) for more discussion of grading practices of GCSEs. 
? Permanent exclusion in England is the same disciplinary action as expulsion in the US’ and other education systems. 
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The second two groups of schools became academies after the UK government changed in the 
election of May 2010 (to a Conservative/Liberal Democrat coalition) in a government drive 
towards mass academisation. In this period, two types of academies came about through 
conversion. Firstly, there was a continuation of the sponsored programme, albeit not featuring such 
disadvantaged schools as prior to 2010 and which remained relatively small scale. Secondly, new 
academies known as converter academies appeared. These formed the vast majority of new 
academy schools, which no longer required a sponsor and which were typically highly performing 
schools. 

The analysis shows that, following academy conversion, academies do exclude more pupils 
enrolled in year 11. The higher rate of exclusion is bigger among the pre-2010 academies. When 
one notes that prior research has reported quite sizable test score performance gains for the pre- 
2010 academies (Eyles and Machin, forthcoming, Eyles et al., 2016a; Eyles et al., 2016b), it then 
becomes a natural question to ask whether the observed performance improvements can be 
explained by higher rates of pupil exclusion. Simulations are undertaken in the paper to ask to what 
extent the gains from pre-2010 academy enrolment can be attributed to permanent exclusion of 
year 11 pupils. This is done by reinstating the excluded pupils in the excluding school and 
calculating a counterfactual level of school performance. In doing this, it is also feasible to quantify 
negative peer effects on other pupils that may have occurred under the counterfactual scenario. The 
simulation exercise suggests that, for the positive estimated impact of academy enrolment on GCSE 
test scores in Eyles and Machin (forthcoming) to become insignificant, each excluded pupil would 
have needed to exert a negative peer effect on the GCSE test scores of all other year 11 pupils in 


the same school and school-year of more than 30 percent of a standard deviation. This simulated 


peer effect appears implausibly large to attribute the performance gains from academy conversion 
to deliberate strategic exclusion of poorly performing pupils. 

Moreover, the academic ability of the excluded children in the autonomous schools is seen 
to be no worse on average than that of excluded pupils in non-autonomous schools. Thus, whilst 
pre-2010 academy schools did exclude more, there is no evidence that year 11 pupils excluded 
from academies were worse performers than year 11 pupils excluded from control schools. Pre- 
2010 academies also permanently excluded more pupils in year 11 that were still allowed to take 
their GCSE exams in the school that excluded them, which removes scope for strategic 
manipulation of GCSE results. Moreover, no systematic association appears between performance 
gains and changes in permanent exclusion following conversion. The schools that experienced the 
largest increases in exclusion rates following conversion are not in fact the same schools that 
experienced the greatest performance gains. 

The overall conclusion which follows is that, rather than being used as a strategic means to 
boost measured school performance, the higher rate of exclusion seems to have been part and parcel 
of the tough discipline procedures that the pre-2010 academies adopted. An analysis of the 
distribution of the estimated treatment effects shows that, although a handful of pre-2010 
academies experienced large increases in their rates of exclusion following conversion, these 
schools were not the drivers of the average performance gains recorded by pre-2010 academies. 
This conclusion is further reinforced by the much smaller impact of academy status on exclusion 
for the two sets of post-2010 academies. These schools were better performing schools prior to 
their academy conversion and did not face the kind of behavioural problems and pressure for 
improvement faced by the pre-2010 academies. This reinforces the overall conclusion that the 


higher rate of permanent exclusion from the poorly performing schools that first converted to 


academies in the 2000s was a consequence of their need to adopt a rigorously enforced disciplinary 
policy (which may have contributed to performance improvements), rather than a means to 
artificially massage and so manipulate their overall performance and league table position. 

The remainder of the paper is structured as follows. In the next section, the legal framework 
in relation to school exclusion in England and its connection to school league table performance 
are described. The data and some aggregate figures on exclusion in secondary schools in England 
are also presented. Section 3 discusses the relevant related literature, and section 4 presents the 
research design. Section 5 presents the main results. Section 6 considers possible results 
manipulation and prior attainment differences for pupils excluded from schools with different 


levels of autonomy. Finally, section 7 concludes. 


2. Pupil Exclusion 


Good behaviour in school has frequently been identified as essential to ensure that all pupils can 
benefit from education (Department for Education, 2015). Indiscipline in school can negatively 
influence the provision of education in several ways. It can disrupt the agenda of teachers and 
hinder their ability to provide pupils with a high quality of education; it can divert schools’ 
resources away from their original purposes; and, more generally, it can contribute to make schools 
an unsafe learning environment for both the school personnel and for the pupils. Indeed, the 
educational output of pupils is likely to be directly affected by their behaviour in school (Lazear, 
2001). In the English setting we study, schools have policies, procedures and staff training in place 
to promote good discipline and inhibit bad behaviour. Such policies are broadly publicised to pupils 


and their families, as well as school personnel, to ensure that they all know what standards of 


behaviour are expected in school and the array of actions that schools can take to tackle bad 
behaviour.? 

One sanction that is available to a school to restore discipline is pupil exclusion. The 
decision to exclude requires a pupil to leave the school premises, and it is a decision that only head 
teachers (principals) have the power to take. According to Department for Education (DfE) 
guidelines, such decision must be “lawful, rational, reasonable, fair and proportionate”. However, 
head teachers are expected to, and often have to, use discretion in taking this decision. In England, 
a pupil can only be excluded on disciplinary grounds, while it is unlawful to exclude a pupil due to 
poor academic performance. Possible reasons for exclusion include persistent disruptive behaviour, 
bullying, theft, physical assault, verbal abuse, threatening behaviour, damage, and drug and alcohol 
related offences. 

In England, the educational setting of focus in this paper, there are three types of exclusions 
that head teachers can use: lunchtime exclusion, temporary exclusion and permanent exclusion. A 
lunchtime exclusion consists of an exclusion from the school premises for the duration of the 
lunchtime period. A temporary exclusion consists of excluding a pupil from the school premises 
for a set period of time, up to a maximum of 45 school days in a single academic year. This sanction 
is deemed appropriate in response to breaches of the school’s behaviour policy, such as disruptive 
behaviour, where these are not sufficiently serious to justify a permanent exclusion and smaller 
sanctions (e.g. detention) are deemed inadequate. 

Permanent exclusion is the most serious disciplinary decision that can be taken in response 


to bad behaviour. It means the pupil has to leave the school premises and s/he will not be allowed 


3 One good example of this is the Student Behaviour Policy 2016-2017 of the Haggerston Community School in the 
Borough of Hackney (London), that can be found at the following link: https:/(www.haggerston.hackney.sch.uk/wp- 
content/uploads/2017/03/Behaviour-Policy-2016-2017.pdf 
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to return to the school. In practice, this decision is usually the final step in a process for dealing 
with indiscipline after a wide set of other strategies have been tried without success. In principle, 
this should count as an acknowledgment by the school that this has exhausted all possible strategies 
to deal with an undisciplined pupil. This decision should only be taken in response to a serious 
breach, or persistent breaches, of the school’s behaviour policy, and where the presence of the pupil 
in the school would seriously harm the education or welfare of the pupil, as well as others’ in the 
school. 

Following a permanent exclusion, schools and local education authorities have to arrange 
alternative provision from the sixth day of the exclusion of pupils of compulsory school age. The 
head teacher must notify the school’s governing body, which has to review the exclusion decision.* 
If the governing body upholds the exclusion decision, and if the parents of the excluded pupil 
dispute the decision, they can also request a review of the exclusion by an Independent Review 
Panel (IRP). The IRP does not have the authority to reinstate a pupil, but it can direct a governing 
body to reconsider its decision.° 

Even though a pupil remains formally enrolled in the excluding school until the end of the 
exclusion process, permanent exclusion (and the subsequent arrangement of alternative education) 
is de facto one way to modify the list of pupils on-roll and, thus, the list of pupils that will contribute 
to the school-level aggregate results at the end of the academic year. Moreover, since inclusion in 
a secondary school’s league table results hinges on the census carried out in the January of a pupil’s 
year 11 school attendance, it is possible that schools that operate more autonomously also within 


the exclusion process might take advantage of this disciplinary sanction. 


4 This is also the case for fixed-period exclusions that would result in a pupil being excluded for more than 15 school 
days in any one term, or missing a public examination. 

> Since 2011/12, IRPs have replaced the Independent Appeal Panels that, unlike IRPs, did have the power to reinstate 
an excluded pupil in school. 


Data on Pupil Exclusion 

The principal source of data that can be used to study pupil discipline in England is the 
National Pupil Database (NPD) of the Department for Education. This large administrative dataset 
contains detailed information on all pupils enrolled in state-maintained schools in England since 
2001/02. For the analysis of this paper, data are used from the academic years 2001/02 to 2014/15 
inclusive, and include all pupils enrolled in secondary school in English state-maintained schools. 
For each pupil, information on a set of relevant characteristics is collected every year (e.g. on age, 
gender, ethnicity, native language, special educational needs status, free-school-meal eligibility, 
and the school where the pupil is enrolled on the census date). Pupil-level information on schooling 
outcomes that include the pupils’ school test scores at different stages of their schooling trajectory 
(Key Stage 2 (KS2) exams, which pupils take at the end of primary school, and Key Stage 4 (KS4) 
exams, the high-stakes GCSE exams that pupils take at the end of secondary school) is also 
available. 

Of particular importance for the current analysis is that the NPD also provides 
administrative records for a set of behavioural outcomes at the pupil level, namely for school 
absence and for the different types of exclusion described above. The NPD data on exclusion 
contains information on the type of exclusion (permanent, temporary, lunchtime), on the reason for 
exclusion, on the start date of the exclusion, on the school where the pupil was enrolled at the time 
of the exclusion and, for temporary exclusions, on the duration of the exclusion. The focus of this 
paper is on the most serious discipline action, permanent exclusion, and in total from the school 
year 2001/02 to 2014/15 there is detailed information on pupil characteristics, school test scores 


and permanent exclusion for 10,675,192 pupils enrolled in state-maintained secondary schools in 


England. Since every pupil is observed for up to five years in secondary school, this results in 
40,471,691 pupil-year observations. 
Descriptive Analysis 

Table 1 shows the distribution of permanent exclusions for pupils in state-maintained 
secondary schools in England from 2001/02 to 2014/15. Panel A shows the count and shares of all 
permanent exclusions in the school year for year 7 through year 11 pupils. Panel B shows the count 
and shares only for the permanent exclusions that occurred in the months prior to the January 
census in each school year (1.e. from September to the exact date in mid-January when the January 
census was conducted every year). 

The reason for showing pre-January numbers separately is an institutional feature of the 
education system that enables possible strategic results manipulation. The January cutoff is used 
because pupils that are excluded in year 11 after this date may contribute towards the secondary 
school league table results of the school from which they were excluded because they are allowed 
to sit the summer exam in the school. Pupils excluded before January are not allowed to do so. 
Thus, this is a strategic time window during which schools may exclude poorly performing pupils 
that could bring down the school average. 

Table 1 shows that over the whole time period for which we have data 79,868 pupils were 
permanently excluded from school, corresponding to 0.2 percent of the universe of students. Of 
these exclusions, 8 percent were pupils excluded in year 7, 19 percent in year 8, 29 percent in year 
9, more than 30 percent in year 10, and 12 percent in year 11. Panel B reveals that only 31,853 
pupils, or less than 0.08 percent of the total, were permanently excluded in the months of the school 
year that precede the January census. Interestingly, in each of the years 7-10 the fraction of pupils 


excluded in the months prior to the January census is well below half the number of pupils excluded 


throughout the year. This is not the case in year 11. Almost 80 percent of pupils that were excluded 
in year 11 between 2001/02 and 2014/15 were excluded prior to the January census. As noted 
above, this is an important cutoff because inclusion in a mainstream secondary school’s league 
table results crucially hinges on which school year 11 pupils are enrolled in when the January 
census is conducted. This cutoff is therefore critical to the notion that strategic permanent exclusion 
might take place. 

Table 2 reports estimates of the determinants of behavioural outcomes for year 11 pupils. 
A series of logistic regressions were estimated and Table 2 reports estimated odds ratios from these 
(with over/under unity corresponding to a higher/lower outcome likelihood). Complete data is 
available on 7,336,508 year 11 pupils starting from 2001/02. Estimates of the likelihood of 
exclusion reported in columns (1), (2) and (3) reveal a consistent profile regarding characteristics 
of pupils with better or worse discipline outcomes. Pupils most at risk of exclusion in year 11 are 
from low socio-economic background (as proxied by the free school meals, FSM, variable), are 
males and are English native speakers. Better prior schooling attainment (Key Stage 2) is also a 
strong negative predictor of year 11 exclusion. 

Columns (4) and (5) report estimates from specifications for unauthorised absence and 
persistent unauthorised absence in year 11. The former is measured by a continuous variable on 
the fraction of school sessions missed without authorisation, while the latter is a binary variable 
that captures whether the pupil missed at least 20 percent of sessions available in the school year.® 
The coefficients in columns (4) and (5) reveal that pupils from low socio-economic background 
are more likely than others to miss school sessions without authorization, as well as to miss 20 


percent or more of the available school sessions. Girls are more likely to be absent from school 


6 This was the Department for Education’s definition of “persistent absence” at the beginning of the period studied 
here. 
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without authorisation, and English native speakers are more likely to miss school without 
permission. Prior academic performance is also a negative predictor of the likelihood to miss school 
without authorization in year 11. 

The assertion that schools operating with more autonomy might have been able to take 
advantage of disciplinary sanctions with an intention to manipulate aggregate school performance 
has been made by detractors and critics of academy schools in England. Indeed, the notion has 
received press and media coverage in recent years.’ The aim of this paper is to rigorously 
investigate this claim. It does so by offering causal evidence of the impact of academy enrolment 
on permanent exclusion of year 11 pupils based on rich administrative data, and then evaluating 
the potential impact this has on school performance differences between more and less autonomous 


schools. 


3. Strategic Manipulation of School Performance 


In England, as well as many other countries around the world, the quality of education is measured 
through standardized testing procedures working through a national curriculum and secondary 
schools are ranked based on their aggregate test scores. This system appears desirable for a number 
of reasons. Lack of accountability can induce teachers to shirk (Angrist et al., 2017a), whereas its 
presence, at least in principle, gives an incentive to schools and teachers to put more effort in the 
provision of education. Test-scoring procedures also provide summary and uniform performance 
metrics that can be used to judge schools and potentially take remedial action. Finally, the 
widespread public availability of school rankings enables parents to compare schools and choose 


among them. 


7 See the Appendix for a number of selected examples of media coverage in the UK and US. 
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Moral Hazard Issues 

This study contributes to the recent literature on the moral hazard that can arise from test- 
scoring procedures and on the various forms of cheating, by both administrators and teachers, 
which may contaminate the reliability and fidelity of standardized test results (see Neal, 2013, and 
Battistin, 2016). In an early study, Jacob and Levitt (2003) present evidence from Chicago public 
schools that test score manipulation by unscrupulous administrators and teachers responds strongly 
to small modifications in incentives, especially in low-achieving classrooms. Jacob (2005) uses 
data from Chicago public schools to show that score accountability pressures may also induce 
teachers to ‘teach to the exam’. Neal and Schanzenbach (2010) show that, in order to inflate their 
assessment scores, teachers may pay less attention to top- and worst-performers in a given grade, 
focus on pupils close to the relevant performance cutoff, and leave worst-performers behind. 

No evidence of test score manipulation is found in Lavy (2009), who shows that a monetary 
incentive programme for English and math teachers in Israel led to greater teacher effort, higher 
test taking rates, improved conditional pass rates, and higher mean test scores. Diamond and 
Persson (2016) estimate the positive and lasting effects of maths grade inflation in the last year 
prior to high school in Sweden. Dee et al. (2017) document regular test score manipulation in New 
York’s high school exit exams, known as the Regents Examinations, which hurts better students 
and helps worse students who are at risk of dropping out. Borcan et al. (2017) also find evidence 
of grade manipulation on high-stakes exams at the end of high school in Romania. Angrist et al. 
(2017a) find evidence of teachers’ cheating in smaller classes in Southern Italy, and they interpret 
this as a form of cheating that is induced by the lack of accountability of teachers in the Italian 


mezzogiorno, rather than an excess of it. Battistin and Neri (2017) document some consequences 
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of score regrading at the end of primary school in England for pupils just below relevant 
achievement thresholds. 

Finally, and of direct relevance to this paper, Machin et al. (2017) look at whether or not 
English secondary school students just pass or fail their GCSE English Language examinations 
(they have data on actual marks and so can study differences of one mark either side of the grade 
C pass cutoff). Their focus is on long term effects of just passing or failing, but part of their analysis 
highlights that the nature of the examination system (especially its external grading system via 
exam boards) makes it hard to teachers in schools to directly manipulate grades in these high stakes 
examinations. Despite this, incentives for teachers and head teachers now play a big role in the 
English education system where the ranking of schools and their managers by pupil performance 
has become a central feature of the system. If direct manipulation of exam results is difficult, then 
this raises the distinct possibility that other means to try to enhance league table positions might be 
attempted. The one considered in depth in this paper is permanent exclusion of low performers so 
that they do not get entered into the exams that count towards league table positions. 

School Accountability 

This paper is therefore most closely related to a literature that investigates how 
accountability pressures can lead schools to try and shape the pool of pupils that sit exams in a 
strategic manner. Figlio and Getzler (2002) find that, following the introduction of the Florida 
Comprehensive Assessment Test in 1996, schools began to label students as disabled and, thus, as 
unable to contribute to the school's aggregate results, especially in low-income schools and for 
pupils with low prior academic performance or from low socio-economic background. Figlio 
(2006) documents that schools in Florida attempt to reshape the pool of test-taking students by 


widening the gap between the punishment assigned to low-performing pupils and to high- 
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performing pupils in periods of high-stakes exams for pupils in testing grades. Jacob (2005) shows 
that, under the accountability regime introduced by the accountability reform in 1996/97 in Chicago 
public schools, teachers started to retain pupils preventively, they increased special education 
placements and they substituted away from low-stakes subjects. Cullen and Reback (2006) also 
find evidence of strategic efforts in Texas schools in the 1990s to manipulate the pool of test-taking 
students, especially affecting low-performing Hispanic and Black students. Finally, Angrist et al. 
(2017b) document enrolment manipulation to produce smaller classes in Israel, plausibly reflecting 
the schools’ efforts to appeal to educators and parents, and attract greater public financial support. 
However, the interest of their paper lies in the effect of class size on school achievement, and they 
do not find this effect to be affected by enrolment manipulation. 
School Autonomy 

The content of this paper also relates to an important, growing literature on school 
autonomy. A number of countries have recently implemented important reforms of their schooling 
systems to introduce new types of schools that have more autonomy to operate than do regular 
state/public schools. Such school reforms have taken place in a number of countries where 
inequality in access to education, as well as concerns with the quality of public education, have 
featured prominently in the public debate. Attempts to improve pupil performance, reduce 
educational inequalities and a quest to try to find the optimal educational structure are all factors 
underpinning the reforms. Over the last 25 years, charter schools have been introduced in the US, 
free schools in Sweden (‘friskolor’), academy schools in England, and ‘Tomorrow’s Schools’ in 
New Zealand. In the belief that increased autonomy would enable schools to deliver a better 
service, these new school types were granted greater autonomy in several dimensions in 


comparison with already existing schools. 
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Charter schools in the US are especially prevalent in large urban areas, such as Boston, Los 
Angeles, New York and Philadelphia where many students are poor and non-white. Schools in 
these areas often record low standardized test scores, high truancy rates, and high dropout rates 
(Abdulkadiroglu et al., 2016). Although this literature has found mixed evidence on the effects of 
charter attendance on academic and labour market outcomes (see, e.g., Dobbie and Fryer, 2016), 
positive effects appear concentrated in urban centres and at schools practicing the “No Excuses” 
model, which stresses behavioural norms and work ethic (Abdulkadiroglu et al., 2011; Dobbie and 
Fryer, 2011; Angrist et al., 2016; Dobbie and Fryer, 2015 and 2016). Benefits from charter 
attendance tend to accrue to less-privileged students (e.g., Gleason et al., 2010), and they appear 
larger in math than in reading test scores (see, e.g., Hoxby et al., 2009; Gleason et al., 2010; Angrist 
et al., 2010; and Fryer, 2014). 

Key contributions to the literature on the English academies studied here are Eyles and 
Machin (forthcoming), Eyles et al. (2016a) and Eyles et al. (2016b)*. These studies focus on the 
first (i.e., pre-2010) batch of academies, and they study the impact of the academy conversion on 
pupil intake, pupil performance at the end of secondary school and post-secondary school 
outcomes. They find that the academy conversion led to an improvement in the quality of pupil 
intake, as measured by the pupil prior schooling attainment, while enrolment of FMS eligible pupils 
dropped. Eyles and Machin (forthcoming) and Eyles et al. (2016b) also find that pupils who 
attended academies performed significantly better in their GCSE exams at the end of secondary 
school, with greater gains accruing to pupils enrolled in an academy for longer and to pupils from 


disadvantaged backgrounds (Eyles et al., 2016b). Academies in London experienced the greatest 


8 Machin and Wilson (2008), Machin and Vernoit (2011) and Wilson (2011) first examined the effects of the pre-2010 
academies programme on the quality of pupil intake. Machin and Silva (2013) first documented positive performance 
effects of academy conversion using KS4 test scores. Clark (2009) is also related to this literature as he studied the 
effect of school conversions to become grant-maintained (GM) schools on pupil performance. 
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gains, a result that echoes the evidence on urban charters in the US (see, e.g., Angrist et al., 2013). 
Academy enrolment raises the probability of completing a Bachelor degree by the age of 21, and 
this effect too appears greater for academies in London (Eyles et al., 2016b). Eyles et al. (2016a) 
show that academy enrolment raises the chances to remain in school at the end of compulsory 
schooling, it has longer run effects by raising the probability of entering higher education in the 
four years after completion of KS4 exams, and it increases the likelihood to enrol in a Bachelor’s 
degree in a non-Russell group university. 

In sum, the existing literature suggests positive effects from enrolment in autonomous 
schools, especially those adopting a “No Excuses” approach. However, if autonomous schools do 
deliver performance improvements, one key research question is whether they do so by improving 
the education levels of children enrolled in the school, or whether their extra autonomy means that 
they are able to manipulate measures of school performance. In the English setting we study, one 
way to do this is to exclude poorly performing students who bring down the school average. The 
reviewed literature suggests that this is a possibility, and this is exactly what the popular press, and 
some politicians and educationalists, have argued that academy schools do.’ This, in fact, is a 
concern that has also been raised in US media about charter schools.!° In this paper, we directly 
study this question, looking at causal estimates of academy status on pupil exclusions and their 
strategic nature, and whether it is indeed possible that the pupil performance improvements seen 


in some academy schools can be attributed to this form of strategic exclusion. 


° Appendix B reports selected examples available online of media coverage. 

1° To our knowledge, there is to date no academic research on expulsions by US charters. The closest research compares 
exit patterns of students in charters with exit patterns of students in traditional public schools, and fails to detect 
significant differences between these (see Zimmer and Guarino, 2013; and Furgeson et al., 2014). 
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4. Autonomous Schools and Research Design 


Autonomous Schools in England 

The secondary sector of the English educational system consists of around 3000 schools spread 
across 152 local education authorities. Up to the late 1990s, most secondary schools in England 
were community schools that were largely under the control of local educational authorities. In 
addition to community schools, religious schools (called voluntary controlled or voluntary aided 
schools) and foundation schools made up the remainder of the state funded secondary sector.'! 
These schools existed in the 1990s and they continue to exist today. Community schools differ 
from other types of schools in a number of ways. For instance, foundation and voluntary aided 
schools usually have more autonomy than community schools. Some also have selective admission 
procedures inherited from the past, whereas a selection system based on ability is not permitted to 
other state-funded schools in England. For community schools, admissions of pupils, as well as 
hiring and firing of staff, are managed by the local education authority. Finally, no state-funded 
school in England is allowed to charge fees to their students. 

Up until 2002, community schools operating under the control of local authorities 
constituted the majority of secondary schools in England. Between the late 1990s and the early 
2000s a general recognition spread in England that some secondary schools were not delivering a 
good enough education to their pupils. These schools were usually concentrated in disadvantaged 
urban areas, and their poor educational standards, coupled with significant discipline problems, 
generated mounting pressure on public authorities. The proposed solution was to replace the 
existing school with a new type of state school - the academy school. The first academy schools 
were introduced by the 1997-2010 Labour government, and they opened their doors in September 


"| During the 1990s, a small number of so-called City Technology Colleges (CTCs) also existed. In some ways these 
were the pre-cursors of academy schools (see, for example, the discussion in West and Bailey, 2013). 
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2002. While only a handful of schools were converted to academies in the school year 2002/03, 
5,272 academies existed by April 2016. In hindsight, the introduction of academy schools in the 
English education system not only constituted an important development in the history of education 
in England, but it also appears to be one of the most radical and encompassing programmes of 
school reforms experienced in recent years across advanced economies. 

Academy schools are independent, non-selective and state-funded schools. They are run 
outside of the control of the local education authority and, in most cases, they are conversions from 
an already-existing predecessor school. Prior to the Academies Act of 2010, when key changes 
were made to the definition of academy status, they were managed by a private team of independent 
co-sponsors. The sponsors of the academy school delegate management of the school to a mainly 
self-appointed board of governors that have the responsibility to employ the academy staff, to agree 
levels of pay and conditions of service, to decide on the policies for career development, discipline, 
performance management and staffing structure, and to manage school admissions.'* Unlike other 
schools, including foundation and voluntary aided schools, academies also have more autonomy 
in a number of dimensions, such as the freedom to opt out and not follow the national curriculum 
(except for core subjects) that defines what is studied in English secondary schools. 

Up until 2010, the central aim behind the creation of these schools was to turn around failing 
schools. In order to establish an academy school, it was necessary to identify a sponsor, which 
could be an individual, a business, a religious body or a university. Together with the school’s local 
authority (LA), the sponsor needed to submit a formal expression of interest to the Secretary of 


State for Education for the ministerial approval and, upon receipt of approval, the feasibility stage 


Although academies have control over their own admission arrangements, they operate under the same admissions 
code as other state schools. Thus, they cannot select students based on ability and they have to prioritise the most 
disadvantaged students, i.e., those who are or have been in care and those with learning difficulties. 
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was the last step before the actual academy conversion of the school.'* Since the first academies 
opened in 2002/3, a growing number of schools converted in the following years and by January 
2010, 203 academy schools, or four percent of secondary schools, were operating in England. 

A major change to the academisation process of the English education system started in 
May 2010, when the new Conservative/Liberal Democrats coalition government was elected and 
quickly introduced the Academies Act of 2010. While the pre-2010 Labour government conceived 
the academies programme as a remedial programme for poorly performing schools, the Academies 
Act of 2010 brought about a step-change to it in order to bring greater autonomy and competition 
in the entire school sector. The programme was no longer limited to poorly performing schools, 
and schools no longer had to sign up a sponsor prior to becoming an academy'*. Furthermore, 
primary schools were also allowed to become academies. The new policy regime resulted in a 
remarkable increase in the number of academies in England; by January 2015, 2,075 secondary 
schools were academy schools, constituting 61% of secondary schools in England. In line with the 
new legislation, and unlike the pre-2010 academies, these were generally high performing schools 
enrolling advantaged pupils, and only few of these (531, including the 203 pre-2010 academies) 
signed up a sponsor prior to conversion (Eyles et al., 2016a). 

Table 3 shows results from permanent exclusion logistic regressions in year 11 by school 
type. Panel A shows estimates for the likelihood of permanent exclusion throughout year 11, whilst 
panel B shows the same for pre-January census permanent exclusion. For purpose of comparison, 


column (1) of panel A reproduces the column (1) of Table 2 results, while column (1) of panel B 


3 The sponsor’s role was mainly to contribute management expertise. Until 2007 sponsors were also required to 
contribute 10 percent to the capital cost of new school buildings, or up to a maximum of two million pounds, but this 
requirement was relaxed in 2007 and removed in 2010. 

'4 Under the new regime, would-be ‘converts’ only had to debate their intention to convert with parents, staff and 
pupils, and obtain permission to convert from the foundation or religious body funding the school. The intention to 
convert then had to be registered at the Department for Education (DfE), and information had to be submitted on school 
attainment and school finances for the last three years, and on the latest inspection by the school inspectorate (Ofsted). 
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does likewise for column (2) of Table 2. These are the all school estimates. In the rest of Table 3, 
estimates are shown for year 11 pupils in the different school groupings: column (2) for pre-2010 
academies after conversion, column (3) for all post-2010 academies, column (4) for post-2010 
sponsored academies, column (5) for post-2010 converter academies and column (6) for non- 
academy or future to be academy schools.’ 

The coefficients reported in Table 3 mostly reveal a consistent pattern both across school 
types and for all year or pre-January census exclusion samples. In particular, pupils from low socio- 
economic background face a greater risk of exclusion in all types of schools. Males and native 
English speakers are also more likely to be excluded from all school types. Individuals with inferior 
prior academic performance are also much more likely to be excluded in year 11 in all types of 
schools. Overall the message to take away is that pupils with similar characteristics are more likely 
to be excluded in each of the school types considered in Table 3. 

Research Design 

The next step is to move on from the descriptive analysis undertaken so far to study the 
causal effect of school autonomy on pupil permanent exclusion in year 11. The research design 
used for this is a difference-in-differences approach that exploits the gradual introduction over time 
of academy schools in England. Due to the important differences between the academies 
programmes before and after 2010, the two are studied jointly as well as separately, with the post- 
2010 analysis also distinguishing between sponsored and converter academies. !° 

Table 4 provides a timeline of academy conversions between 2001/02 when there were no 


academies and 2016/17, for schools with complete data available from the start year. As Table 4 


'S Depending which school year the data covers. 
'6 For some empirical evidence on the differences between pre- and post-2010 academies, see Eyles et al. 
(forthcoming). 
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shows, the growth of academies began slowly — just three academy schools opened in 2002/03 — 
rising gradually to 208 schools by 2010/11, and a very rapid growth followed in the period of mass 
academisation. 

As with other work in the area of academy effects on pupil outcomes (notably Eyles and 
Machin’s, forthcoming, analysis of pupil performance effects of academy conversion for the pre- 
2010 academies), there is a need to compare academies with a set of similar schools. The approach 
taken here is as in that other work, namely to use schools that will become academies in due course, 
but after the sample period ends. These have been shown to be well matched on pre-conversion 
observable characteristics (balancing tests for our samples of schools are given in Appendix Tables 
B1 to B4, and will be discussed in due course), but will also capture unobservable common reasons 
for conversion to an academy (like ethos or aspiration to become an academy). 

As Table 4 shows, for pre-2010 academies by January 2009, 94 schools had become 
academies. These form the treatment group for the pre-2010 analysis, and the matched control 
group is the 114 schools that became academies in the school years 2009/10 and 2010/11. 
Importantly, all of these were approved for conversion prior to May 2010, i.e., under the Labour 
government and before the coalition government was elected in 2010. 

Table 4 also shows that, for the analysis of the post-2010 academies programme which 
involved many more academy conversions, there is complete information on 1,474 academies, i.e., 
on 1,220 converters and on 254 sponsored academies. The research design adopted here defines 
treatment schools as becoming academies following the 2010 general election and by January 2014. 
The control group includes schools that became academies after this date. Thus, for the post-2010 
sponsored academies programme, there is a treatment group of 157 schools that became sponsored 


academies by the school year 2013/14, and a control group of 97 schools that became sponsored 
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academies later. For the post-2010 converter academies, there is a treatment group of 1,116 schools 
that became converter academies by the school year 2013/14, and a control group of 104 schools 
that subsequently became converter academies. When the two programmes are analysed together, 
all the treatment schools from the separate programmes are combined and compared with all the 
control schools from the different programmes. This results in a treatment group of 1367 treatment 
schools to be compared with a control group of 315 schools. 

The modelling approach of Eyles and Machin (forthcoming) is modified here to look at 
pupil exclusion as the outcome of interest rather than performance. Since academy conversion can 
change the ‘quality’ of pupil intake, an ordinary least squares regression of exclusion on academy 
status is unlikely to yield a credible estimate of the impact of being enrolled in an academy on 
exclusion. This problem of composition change in the school is circumvented by focussing 
attention only on pupils already enrolled in the school prior to conversion. This is the approach 
taken in Abdulkadiroglu et al. (2016) and Eyles and Machin (forthcoming), where pre-enrolled (in 
the respective studies referred to as ‘grandfathered’, or ‘legacy-enrolled’) pupils in schools that 
later became charters in the US, or academies in England, are studied. 

Let t indicate academic year, let s(i, t) indicate the school in which pupil i is enrolled in 
year t, and let g(i, t) indicate the grade in which pupil i is enrolled in year t (this takes values 7-11, 
i.e., the secondary school years in England). The year of conversion for school s is denoted by CYs. 
Finally, conversion cohorts are sets of academy schools — At — that convert in the same academic 
year t. Academy status is a binary variable and equals 1 for pupils enrolled in an academy at the 
start of year 11. Academy conversion is defined as: 


Lift > CY.) 


Academyit = . ift < CY¥sa 7 
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The basic difference-in-differences set up for a given conversion cohort of year 11 pupils 1 

in treatment and control schools is: 
EX;,.= f(s, t) +0, Academy;+ ) BijXjit+ Oi Pe +Viit (2) 
j=l 

where EX is a binary variable that takes value | if a pupil was excluded in year 11 and 0 otherwise, 
f(s, t) is a general (to be specified) function of school and time, X is a set of pupil level control 
variables, P is pupil level prior achievement and v1 is an error term.'’ Each cohort of treatment- 
comparison estimates can be stacked to obtain average difference-in-differences estimates of the 
coefficients in equation (2). 

For this set up, one can define legacy enrolment status for pupil i in an academy school as 
an intention to treat binary variable ITTit that equals one for those legacy enrolled in academy 
schools and 0 for pupils legacy enrolled in control schools: 


1 if s(it-1)EA, and 11>g(i,t-1)=>7 (3) 
0 if [s(i,t-1)€A, and g(i,t-1)=11] U s(i,t-1)€A, 


ITTn= 

Pupils are legacy enrolled (i.e., ITTit = 1) if they are enrolled in an academy in the year 

prior to conversion and are not in their final year of compulsory schooling. This remains fixed 
through time spent in secondary school. Since pupils spend five years in secondary school, 
exclusion effects in year 11 for pupils who stay in the school for a maximum of four years post 
conversion, including the year of conversion itself, can be estimated. This exercise can generate a 


causal effect for year 11 pupils because they had already previously enrolled in the same 


(predecessor) school before it converted to become an academy. 


'’ The control variables in X are gender, free-school meal eligibility status, ethnicity, whether a pupil is native English 
speaker and a proxy for pre-determined school-level behaviour in the ITT school where pupil i pre-enrolled. Prior 
achievement P is measured by the end of primary school Key Stage 2 performance of the pupil. 
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There is, however, still a problem of estimating equation (2) only for the sample of legacy 
enrolled pupils. The issue that arises is that not all students initially enrolled in a school that 
converted to an academy (ITTit = 1) remain in the school until the start of year 11 (Academyit= 1). 
Thus, ordinary least squares estimates of 01 from (2) estimated on legacy enrolled pupils may not 
reflect a causal estimate if the decision to defy ITT assignment is non-random. Selection into and 
out of treatment can however be addressed by using intention to treat status (ITTit) as an instrument 


for Academyit, to estimate a local average treatment effect (LATE) as follows: 


J 
Academy;,=a,+O,+92ITT+ > BajXjie+ P2Pe+ Voit (4) 
j=l 


EXj,=A,+Q,+O3ITTi+ ) B3jXjict+ O3Pic+ Voit (5) 
j=l 


where f(s, t) is specified as the additive function a, + a,, with a, being a set of school fixed effects 
measured at the time of legacy enrolment and a; a set of time fixed effects. In the first stage (4), 
estimates of 02 show the proportion of the ITT group that stay in the academy and are still enrolled 
at the beginning of year 11. Equation (5) is the reduced form regression of EX on the instrument. 
The instrumental variable (IV) local average treatment effect (LATE) estimate is then defined as 
the ratio of the reduced form coefficient to the first stage coefficient, 03 /@2. 
Descriptive Difference-in-Differences 

Table 5 shows the results from a descriptive difference-in-differences exercise comparing 
year 11 permanent exclusion three years before and after academy conversion in treatment group 
schools and control group schools. These are shown separately for pooled cohorts of pre-2010 
academies and post-2010 academies, and their respective comparison schools. Results are also 


presented separately for post-2010 sponsored academies and post-2010 converter academies in the 
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third and fourth panels of Table 5. Thus, for example, for schools that became academies in 
2005/06, the pre-period rate of permanent exclusion in year 11 was calculated in the school years 
2002/03 to 2004/05, and the post-period rate of permanent exclusion in year 11 was calculated in 
the school years 2005/6 to 2007/8. These rates were compared with the exclusion rates in the same 
school years of control group schools. 

Considering first pre-2010 academies in panel A of Table 5, column (1) reveals that 0.18 
percent of pupils enrolled in year 11 in pre-2010 academies were permanently excluded in the three 
years prior to conversion, and that this rose sharply to 0.35 percent in the three years after academy 
conversion. There is much less change in control schools. By taking the difference between these 
‘pre’ and ‘post’ exclusion rates, and then differencing across these yields the difference-in- 
differences (DiD) estimate shown in column (3). This is sizable at 0.16 percent for the pre-2010 
academies, showing that they did engage in more permanent exclusions after becoming academy 
schools. This statistically significant increase represents a discrete increase in the three post- 
conversion years. 

The analysis of the exclusion rates for the post-2010 sponsored academies, displayed in 
panel B of Table 5, does not exhibit the same pattern. Post-2010 academies display higher rates of 
permanent exclusion in year 11 both before and after conversion in comparison with control 
schools. The unconditional DiD comparison for these schools reveals that their permanent 
exclusion rates in year 11 increased by 0.007 percentage points compared to control schools. 
However, the DiD estimate for post-2010 academies is not significantly different from zero. 

The analysis of the exclusion rates for the post-2010 sponsored academies shows that these 
schools, unlike their control schools, excluded slightly more pupils after they became academies. 


However, also in this case the DiD estimate is not significantly different from zero. Post-2010 
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converter academies display higher rates of permanent exclusion in year 11 compared to control 
schools both before and after academy conversion. However, the unconditional DiD comparison 
between treatment and control post-2010 converters does not indicate any significant difference in 
the exclusion rates in year 11 between these two groups, as both experienced a similar decline over 
time in the rates of exclusion in year 11. 

Scope for Strategic Manipulation 

The earlier discussion of the numbers in Table | revealed that almost 80 percent of pupils 
that were permanently excluded in year 11 from 2001/02 to 2014/15 were excluded earlier in the 
school year than the January census. This does not reflect seasonality in bad behaviour of pupils, 
given that in each of the years 7-10 the fraction of pupils excluded in the months prior to the census 
was well below half the number of pupils excluded throughout the year. Moreover, as discussed in 
section 2, the January census is an important cutoff for the purpose of inclusion in a mainstream 
secondary school’s league table. 

The very different year 11 within-school year profile of permanent exclusions raises the 
question as to whether the pre-January higher rates are observed because schools may be trying to 
manipulate their end of final year exam results. This is even more pertinent for the pre-2010 
academies whose permanent exclusion rates went up after they became academy schools. In the 
data being analysed in this study, it is possible to study this as we are able to track in which schools 
permanently excluded pupils sit their GCSE exams. 

Table 6 shows the fraction of pupils that sit exams in the excluding school by period of 
permanent exclusion within year 11.'* Columns (1) and (2) show results for all schools, and 


columns (3) and (4) for the matched sample of treatment and control schools. Two interesting facts 


18 Tables Al and A2 in the Appendix show the fraction of pupils that sit GCSE exams in the excluding school by 
month of permanent exclusion in year 11. 
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emerge. First, Table 6 reconfirms that the vast majority of year 11 permanent exclusions - roughly 
80 percent — occur before the January census. Since this is unlikely to reflect seasonality in bad 
behaviour, this is suggestive that schools may be trying to shape the pool of KS4 takers by 
excluding more pupils at the beginning of year 11. 

The second striking result of Table 6 also concerns the hypothesis that schools may use 
permanent exclusion in year 11 in a strategic manner. The results in Table 6 very strongly show 
that pupils who are permanently excluded before the January census are very unlikely to sit their 
summer exams at the end of year 11 in the school that excluded them. While almost 70 percent of 
pupils that are excluded after the January census are allowed, in spite of the exclusion, to sit exams 
in the excluding school, the comparable figure for pupils excluded prior to the January census was 


.!? Thus, schools may well use permanent exclusion at the beginning of year 11 


around 10 percent 
in a strategic way to manipulate results. For the remainder of the paper, pupils excluded in year 11 
prior to the January census are referred to as strategically excluded pupils. 

Table 7 shows descriptive DiD results in the same way as the earlier Table 5 did for any 
permanent exclusion, now placing the focus on this measure of strategic permanent exclusion. The 
analysis of pre-2010 academies, shown in panel A of Table 7, reveals a positive and statistically 
significant increase in strategic permanent exclusion taking place after academisation relative to 
matched future to be academy control schools. By contrast, the analysis of post-2010 academies 
fails to detect an overall increase in year 11 strategic permanent exclusions. The DiD estimate for 


post-2010 converters appears positive and significant, but also much smaller than the DiD estimate 


for pre-2010 academies and not robust to the inclusion of post-2010 sponsored academies in the 


19Tt is important to notice that, although some of the excluded pupils are able to sit exams in the excluding school, 
almost all have to leave the school premises, and only if they are reinstated, they are subsequently allowed to return to 
the school. 
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analysis. This descriptive evidence suggests that academies, especially those that opened prior to 
2010, may have started to exclude more following conversion, and they may have done so in a 


strategic manner in order to manipulate the pool of KS4 takers at the end of year 11. 


5. Statistical Results 


This section turns to the main statistical estimates that look at the causal effect of academies on 
permanent exclusion in year 11 and that subject the descriptive findings to more stringent statistical 
testing through a research design specification that accounts for endogenous sorting of pupils 
across schools. The point of departure is to show that the choice of control schools is justified in 
terms of pre-academisation balancing tests, followed by reporting of results from the main 
statistical analysis. 
Balancing Tests 

Tables B1 to B4 in the Appendix report balancing tests comparing pupil and school 
characteristics for pre-2010 academies, post-2010 academies, post-2010 sponsored academies and 
post-2010 converter academies to their respective sets of control schools. The tests were calculated 
just before the beginning of the pre-2010 and post-2010 programmes, in 2001/02 as reported in 
Table B1 for the pre-2010 academies, and for 2009/10 as shown in Tables B2, B3 and B4 for post- 
2010 academies. In both cases, these dates precede any academy conversion under each academies 
programme that is studied. 

For pre-2010 academies, the comparison between treatment and control schools shows 
them to be balanced almost in all of the dimensions considered. The one exception is percent white, 
which is lower in pre-2010 academy schools (most likely reflecting the well-known fact that more 


such schools were located in urban, inner city locations). Nevertheless, the comparisons as a whole 
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clearly justify the definition of the control group for the analysis. The picture is not quite so strong 
for the post-2010 academies, although it is important to note that all the variables considered in the 
balancing tests are included as covariates in the reported statistical analysis. Post-2010 academies 
seem to enroll fewer free school meal eligible pupils compared to their set of control schools. When 
post-2010 sponsored and converter academies are analysed separately, the presence of free school 
meal eligible pupils appears well balanced between treatment and control schools but the prior 
achievement Key Stage 2 score appears respectively higher for converter academies (as shown in 
Table 4.C) and lower for sponsored academies (as shown in Table 3.C). The remaining pupil and 
school characteristics prove to be well balanced at baseline. 
Initial Estimates 

Table 8 shows the first set of estimates of the causal effect of academy enrolment on year 
11 permanent exclusion. The Table shows results for pre-2010 sponsored academies, post-2010 
sponsored academies and post-2010 converter academies when these are compared together with 
their respective control groups. Results are displayed for both any permanent exclusion in year 11 
and for strategic permanent exclusion in year 11, i.e., that occurred prior to the January census. For 
each dependent variable, it first shows OLS estimates of equation (1), then ITT estimates and 
finally IV estimates. While the OLS estimates do not account for endogenous sorting of pupils 
across schools, the ITT estimates are based on the legacy enrolment decision that was made prior 
to the academy conversion, and the IV estimates instrument enrolment in an academy in year 11 
with the ITT legacy enrolment status. Thus, the ITT and IV estimates circumvent the issue of 
endogenous sorting of pupils across schools and correct for the potential bias generated by it. 
Overall, the results indicate that academies started to permanently exclude more in year 11 


following conversion. This conclusion holds true whether we assess the effect of academy 
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enrolment on any permanent exclusion in year 11 or on strategic permanent exclusion in year 11. 
This conclusion confirms the argument put forward by some popular press in England on the 
increase in exclusion rates in English academies. 

Table 9 shows estimates of the effect of academy enrolment on the likelihood to experience 
any permanent exclusion in year 11 for pre-2010 academies, post-2010 academies, and separately 
for post-2010 sponsored and post-2010 converter academies. For each academy grouping, it still 
shows OLS estimates of equation (1), then ITT estimates and finally IV estimates. The results for 
pre-2010 academies confirm the conclusions from the unconditional DiD estimates of the previous 
section, as they show that the likelihood of a pupil experiencing permanent exclusion in year 11 
increased in these schools following conversion. In contrast with the conclusions from the 
unconditional DiD estimates of the previous section, a significant increase appears visible also in 
post-2010 academies: although the estimated impact of enrolment in a post-2010 sponsored 
academy is not statistically significant, the coefficient associated to these schools, to post-2010 
converters and to post-2010 academies overall are very similar.”° Although the estimated impact 
of enrolment in a post-2010 academy is much smaller than the estimated effect for the pre-2010 
academies, this effect is statistically significant at all conventional levels. 

This conclusion that there was differential exclusion of year 11 pupils in both pre-2010 and 
post-2010 academies is further corroborated by Figures 1 and 2, which display event study IV 
estimates of the impact of academy conversion on permanent exclusion in year 11 for pre-2010 
and post-2010 academies respectively. The event study generates separate estimates of academy 
impact years since conversion, but also allows examination of possible pre-conversion differences 
between treatment and controls. There are two notable features in Figure 1. First, there seem to be 


2° We could not reject the null hypothesis of equality of the estimated IV coefficients for the different groups of post- 
2010 academies in Table 9. 
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parallel pre-conversion trends between pre-2010 academies treatment and control schools. Second, 
there is a steep and persistent increase in the likelihood of a pupil experiencing permanent exclusion 
in year 11 starting from the year of conversion. Figure 2 also reveals parallel pre-conversion trends 
between post-2010 academies treatment and control schools, though the estimated effect is smaller 
in this case. 

Since the potential use of permanent exclusion for strategic manipulation of GCSE results 
is of central interest to this study, Table 10 shows the estimates of the causal effect of academy 
enrolment on year 11 strategic permanent exclusion separately for each group of academies. Using 
the same Table structure as the previous Table 9 for any permanent exclusion, Table 10 shows the 
OLS, ITT and IV estimates by academy group. The results in Table 10 are consistent with those in 
Table 9, as they also show that academies started to exclude more after conversion. 

Following academy conversion, pupils enrolled in pre-2010 academies became more likely 
to be strategically excluded. The OLS coefficient in column (1) shows that, among pupils in pre- 
2010 academies, academy conversion increased the probability of strategic exclusion by 0.260 
percentage points. The ITT coefficient in column (2) indicates that the academy conversion 
increased the likelihood for legacy-enrolled pupils to get strategically excluded in year 11 by 0.262 
percentage points. Column (3) shows the IV estimate of the effect of academy enrolment on 
strategic permanent exclusion. Since, as suggested by the reported first stage coefficient of 0.929, 
not many pupils switch school, the ITT and IV coefficients are similar. The IV estimate indicates 
that the academy conversion in pre-2010 academies led to a 0.282 percentage points increase in 
the probability that a pupil enrolled in a pre-2010 academy was strategically excluded. Table 10 
also shows results for the post-2010 academies, where an increase in the incidence of strategic 


permanent exclusion following academy conversion could also be detected. Also in this case, very 
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similar coefficients were estimated for post-2010 sponsored and post-2010 converter academies, 
with the estimated effect of academy enrolment being smaller than for the pre-2010 counterpart 
but nonetheless positive and significant. 
Discussion 

The results of Table 10 are inconclusive on whether school autonomy led academy schools 
in England to employ permanent exclusion as a tool to bolster their aggregate test performance. On 
the one hand, the fact that there is an effect for both pre-2010 and post-2010 academies suggests 
that more autonomous schools did use discipline sanctions to manipulate their results. On the other 
hand, the effect appears much larger for pre-2010 academies. As we already discussed, these were 
poorly performing schools prior to academy conversion, they had serious discipline problems and 
they recorded very poor aggregate academic results.7! They became academies precisely as part of 
a remedial programme by the 1997-2010 Labour government. In contrast, post-2010 academies 
were much better schools prior to the academy conversion. Therefore, it is very unlikely that these 
schools faced the same extent of discipline problems. The fact that we find a greater increase in 
permanent exclusion in year 11 for pre-2010 academies compared to the result for post-2010 
academies suggests that the increase in exclusion rates observed in Tables 9 and 10 may simply 
reflect the discipline problems that academies faced, with the pre-2010 academies facing more 
serious discipline problems than the post-2010 academies. 

Moreover, one may be worried that by restricting the analysis to permanent exclusion in 


year 11 and not looking at permanent exclusion in prior school years, this rules out an important 


21 As an example for one of the early pre-2010 academies in our sample, the London Evening Standard in 2003 reported 
that “Supply teacher Kate Gibbs described a nightmare two weeks during which she was confronted by a group of 16- 
year-old boys and thrown across a desk after she tried to stop a violent struggle. She was also regularly sworn at by 
pupils and unfairly accused of racism, a ploy used by rowdy students to duck responsibility for disruptive behaviour. 
And she was shocked by the levels of illiteracy among pupils and by the despair of teachers who appeared to have 


"given up".” The full article is available at the following link: https://www.standard.co.uk/news/super-school-fails- 
after-6-months-6351882.html 
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channel through which academies may attempt to shape the pool of KS4 takers, as academies may 
possibly exclude more also before a pupil reaches year 11. Looking at pre-2010 academies, for 
which we find larger effects, the comparison of the OLS coefficient in column (1) with the ITT and 
IV coefficients in columns (2) and (3) is informative, as it reveals these coefficients to be very 
similar. The OLS appears slightly smaller than the ITT and IV coefficients, but all the estimated 
coefficients are positive, significant at all conventional levels and of similar magnitudes. The 
smaller size of the OLS coefficient suggests that some pre-enrolled badly behaved pupils may have 
left the academy after conversion and prior to the start of year 11. However, the degree of similarity 
between the OLS, ITT and IV coefficients clearly suggests that at best this could only involve a 
small number of pupils, and that most of the strategic exclusion that these schools implement takes 
place in year 11. In turn, this does suggest also that permanent exclusion may have been used by 
these schools to shape their pool of GCSE test-takers. It is also important to notice that the first 
stage coefficient in column (3) indicates that only roughly seven percent of pre-enrolled pupils 
moved to a different school by the time they were in their last year of secondary school. 

Finally, there is the issue that permanent exclusion is a rare event involving very few pupils. 
From an econometric standpoint, exclusion operates in the tail of the distribution of pupils and 
raises issues that a linear probability model may predict outside the 0-1 interval. However, as both 
the endogenous variable of interest and the instrumental variable are binary variables, the 
comparison is de facto one looking at averages of treated and controls in the probability that 
strategic exclusion occurs after academy conversion, holding constant all other covariates in our 
specification. There would be a concern linked to rarity of the outcome variable if a linear 
probability model were used to estimate the marginal effect of a continuous treatment. In this case, 


moving along a linear or a non-linear function in the tails may generate different results. 
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6. Possible Results Manipulation and Exclusion by Prior Achievement 


The key question that emerges from the empirical work presented so far is whether the 
higher exclusion propensities in academies occurred strategically in attempts to manipulate overall 
test results. As mentioned earlier, Eyles and Machin (forthcoming) and Eyles et al. (2016b) 
reported evidence of a positive causal impact of academy enrolment on GCSE achievement of 
pupils in pre-2010 academies. In this paper, the interest is whether this can be attributed to higher 
rates of strategic permanent exclusion. As noted earlier, this is exactly what some popular press, 
some politicians and some educationalists have argued. 

Table 11 shows estimates that replicate the findings of Eyles and Machin (forthcoming) on 
the effect of academy attendance on end of secondary school pupil performance. The outcome of 
interest in Table 11 results is now the (standardised) test score of pupils in the KS4 high-stakes 
exams that pupils sit at the end of year 11.”” In the same way as in Eyles and Machin (forthcoming), 
academy (and control school) enrolment status is based on the January census. Thus, strategically 
excluded pupils are not in the excluding school, and rather appear in whatever new school they 
attend following exclusion.” 

The results in columns (1) to (3) of Table 11 are very much in line with those of Eyles and 
Machin (forthcoming). According to the IV specification in column (3), pupils enrolled in pre- 
2010 academies scored 0.104 of a standard deviation (0.1040) higher than pupils enrolled in 


counterfactual schools. Analogous results for post-2010 academies are also reported, and here there 


2 The data description part of the Appendix describes the points system and measurement of the KS4 test score in 
detail. 

3 Recall that, for the analysis of the exclusion impact of academy enrolment, pupil attendance was defined in terms of 
the school at the beginning of year 11. 
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are opposite conclusions, as no performance improvement could be detected for pupils enrolled in 
post-2010 academies”. 

The results in Table 11 suggest quite clearly that pre-2010 academies recorded a steep 
performance improvement following conversion, whereas post-2010 academies did not record an 
average performance improvement following conversion. The question of whether strategic 
exclusion in year 11 may explain the performance improvement of pre-2010 academies is 
considered directly in this section of the paper, where four complementary pieces of evidence on 
the question are presented. The first is a simulation exercise that constructs a counterfactual 
outcome looking at what school performance would have been had the excluded pupils not been 
excluded during year 11. The second asks whether excluded pupils differed in terms of prior 
achievement in autonomous and non-autonomous schools. The third asks whether autonomous 
schools excluded more year 11 pupils enrolled in the school in the January census, for whom there 
is not scope for strategic exclusion as they are permitted to sit their GCSE exams in the excluding 
school. The fourth asks whether the schools that experienced the greatest increase in exclusion 
rates following conversion are the ones that also experienced the greatest school performance gains. 
It turns out that none of these are in line with the notion of differential strategic results manipulation 
by autonomous schools. Rather, they appear to be symptomatic of genuine discipline issues faced 
by the pre-2010 academy schools. 

Possible Results Manipulation 
To define a counterfactual scenario that enables computation of what the performance effect 


of academisation would have been had strategic exclusion not taken place, a simulation exercise 


4 We do not present separate results for post-2010 sponsored and post-2010 converter academies in Table 11 because 
we retrieved very similar results for permanent exclusion for both groups of post-2010 academies as shown in Section 
5. 
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was undertaken to reinstate strategically excluded pupils back into the excluding schools. There 
are several ways in which this can be done to recompute what school performance would have 
been had pupils not been excluded. Results are shown in Table 12, for the pre-2010 academies 
where there is evidence of differential strategic exclusion. 

The first column of Table 12 reproduces the 0.1040 performance effect from the KS4 
regression reported in Table 11. Column (2) shows what happens to this estimated effect when 
strategically excluded pupils are reinstated in the excluding school. In doing so, the excluded pupil 
is given the lowest GCSE test score (zero) in both treatment and control schools, and then the 
estimated academy effect is reconstructed.*> As column (2) shows, and as would be expected 
because academies exclude more pupils, the estimated performance improvement falls under this 
counterfactual, dropping down ever so slightly to 0.1030. The drop is tiny and reflects that the low 
rate of permanent exclusion cannot in and of itself impact substantively on the overall KS4 impact 
of academy attendance. 

Of course, the counterfactual in (2) increases the number of pupils to higher than it was in 
reality for the GCSE exams. There are two issues to do with this. First, year 11 size is bigger. 
Second, if the excluded pupils had stayed in the school, and their discipline problems prevailed, 
they may have had a negative effect on other pupils. To consider the first of these, in column (3), 
to maintain the actual size of the year 11 cohort, an equal number of pupils were dropped from the 
counterfactual calculation undertaken in column (2). To maximise the performance impact, an 


equal number of top KS4 performers within each school and school year was eliminated, and then 


25 Zero is the worst possible KS4 outcome, and this KS4 outcome is assigned to excluded pupils to simulate the 
hypothetical worst-case scenario that may have induced a head teacher to exclude strategically a poorly performing 
pupil. However, we also looked at a range of other possibilities. One is to reallocate the pupils to the lowest score of 
all other pupils taking GCSEs in the school. A second (albeit endogenous) possibility is to use their actual GCSE test 
scores in the new school where they sit their exams. Using these, in place of zero, made little difference in practice. 
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the academy effect recomputed. To be clear, for example if five pupils were strategically excluded 
from school s in school year t, they were reinstated and the top five KS4 performers in the same 
school and school year were removed. When this is done, as column (3) shows, a further small 
drop, to 0.1020, is seen. 

Thus reinstatement of permanently excluded pupils alone barely affects the conclusion that 
the pre-2010 academies improved end of secondary school performance. But what of peer effects? 
The remainder of Table 12 considers this possibility, by producing counterfactual academy effects 
on pupil performance for increasingly larger negative peer effects that could have occurred had 
excluded pupils not been excluded from treatment and control schools. The counterfactuals are 
constructed with the intention of being able to answer the question, how big would negative peer 
effects have to be to eradicate the performance gains that pre-2010 academies generated? 

The remaining columns of Table 12 (columns (4) through (12)) make it very clear that 
negative peer effects resulting from the small number of permanently excluded pupils would have 
to be implausibly large to cancel out the performance gains. Moving from column (4) which 
assumes a 5% of a standard deviation negative peer effect through to column (12) which assumes 
a huge 50% of a standard deviation negative peer effect does gradually lower the estimated 
academy impact on KS4. But, for a ten percent significance level cutoff, the effect only becomes 
insignificantly different from zero when column (9), with an assumed 35% negative peer effect, is 
reached. And even there the estimated academy effect is still 0.0630, as compared to the starting 
point of 0.1040 in column (1). Thus negative peer effects would have to be implausibly large for 
strategic permanent exclusion to explain away the GCSE performance gains made by pre-2010 


academies. 


37 


Prior Achievement Differences 

The second issue would be if academies demonstrably excluded worse performers. Thus 
there would be a bigger differential performance gain accruing to the autonomous schools. One 
reason why this seems unlikely is that the same result on a need for implausibly large negative peer 
effects shown in Table 12 for treatment and control schools emerges if the control schools are 
dropped from the analysis.”° Columns (1) and (2) in Table 13 also present further evidence in line 
with this, presenting evidence in two different ways that there do not appear to be differences in 
the prior achievement levels of permanently excluded pupils in treatment and control schools. 

In Column (1) of the Table, an interaction between academy attendance and prior 
achievement measured by end of primary school Key Stage 2 performance of pupils is added to 
the IV LATE model of the probability of strategic permanent exclusion that was presented earlier. 
The interaction term is not statistically different from zero, suggesting that the causal LATE impact 
of academy attendance on exclusion does not systematically differ with prior achievement. 

In Column (2) of the Table only the excluded pupils in treatment and control schools are 
considered, and the reported estimate considers whether KS2 test scores differ between those 
excluded from academies and control schools. As the estimate shows, there is no significant 
difference between the prior schooling attainment of pupils excluded in year 11 in academies and 
the prior schooling attainment of pupils excluded in control schools. 

Non-Strategic Permanent Exclusion in Year 11 

The third analysis of this section considers whether pre-2010 academies also excluded more 
pupils in year 11 that were permitted to sit exams in the school that excluded them because their 
date of exclusion post-dated the January census. Of course, such permanent exclusions are outside 


26 The column (1) estimate in this case was 0.0840, and as with Table 12, similarly implausibly large negative peer 
effects would need to operate to offset this performance effect. 
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the strategic time window for pupils’ exam results to not appear in the published school league 
tables. For this empirical exercise, the dependent variable now takes the value one for pupils who 
were permanently excluded in year 11 after the January census and zero otherwise. Absence of 
significance in these estimates would be consistent with the notion that pre-2010 academies only 
excluded more pupils in the strategic window - i.e. those that could be removed from their on-roll 
lists of GCSE test-takers - pointing towards the use of permanent exclusion in year 11 as a means 
for strategic manipulation of aggregate school results. However, the result of this exercise is 
displayed in Column (3) of Table 13 and, as the estimate shows, this is not what is found. There is 
still an academy differential in permanent exclusion post January census. Pupils enrolled in pre- 
2010 academies were more likely than pupils in counterfactual schools to experience permanent 
exclusion in year 11 also after the January census of their year 11’s year and, thus, regardless of 
whether they contributed or not to the school aggregate results. 
School-Specific KS4 Gains and School-Specific Strategic Permanent Exclusion in Year 11 

The fourth part of this section investigates whether the pre-2010 academies that experienced 
the largest increase in strategic exclusion are also the ones that experienced the greatest school 
performance gains, as measured by the Key-Stage 4 test scores of their on-roll list of pupils. To 
this end, school-specific IV estimates of the effect of academy attendance on strategic exclusion in 
year 11 and on KS4 outcomes were retrieved and plotted against each other. Figure 3 shows the 
result of this exercise for pre-2010 academies, and three notable features appear. 

First, while almost the entirety of pre-2010 academies recorded greater strategic exclusion 
rates after conversion, only a handful of schools increased their strategic exclusion rates by more 
than | percent. These may be, in fact, the schools that attracted media coverage and then catalysed 


public attention. Second, there is a weak positive (and statistically insignificant) correlation 
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between increases in strategic exclusion rates in year 11 and KS4 gains after conversion. Third, the 
schools that experienced the largest increases in exclusion rates do not seem to drive the average 
KS4 gains of pre-2010 academies, as these are not the schools that experienced the greatest KS4 
gains after conversion. 

Taken together, the results of Tables 12 and 13 and Figure 3 appear inconsistent with the 
notion that pre-2010 academies may have started to exclude more to strategically manipulate their 
aggregate school performance results. The greater exclusion rates in academies after conversion 
reflect the “No Excuses” approach and its emphasis on discipline and good behaviour in school 
that some pre-2010 academies adopted, but they did not constitute an attempt to manipulate the 
school’s aggregate results.”’ Sizable negative peer effects need to be assumed for the performance 
improvements of pre-2010 academies to become insignificant, and prior schooling achievement 
appears to be unrelated to the effect of academy enrolment on permanent exclusion in year 11. 
Unlike some previous studies that found the use of sanctions and other means of shaping the pool 
of test-taking pupils to be concentrated among poor-performing pupils (e.g., Figlio and Getzler, 
2002, and Figlio, 2006), these results suggest that pre-2010 academies started to exclude more 
following conversion, but this did not relate with pupils’ prior academic performance. Moreover, 
the greater likelihood to experience permanent exclusion in year 11 in pre-2010 academies also 
extends to pupils who contributed to the school aggregate results in spite of the exclusion. Finally, 
although some schools did experience sizable increases in strategic exclusion in year 11 following 


conversion, these are not the schools that recorded the greatest performance gains among pre-2010 


27 An example of the tough discipline policies of the pre-2010 academies is the Student Behaviour Policy 2015 of the 
Mossbourne Community Academy in the Borough of Hackney (London). This was one of the first pre-2010 academies 
to open in England and its Student Behaviour Policy 2015 is accessible here: http://www.mca.mossbourme.org/wp- 
content/uploads/sites/3/2015/08/WA-4-3-Behaviour-inc-sixth-form-H-A-A greement.pdf 
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academies. In contrast, performance gains appear unrelated to increases in exclusion for pre-2010 


academies. 


7. Conclusion 

A number of countries have recently reformed their schooling systems by introducing new types 
of schools that have more autonomy to operate than do regular state/public schools. The most well- 
known, and studied, are charter schools in the US and academy schools in England. A recent 
literature has documented significant gains from enrolment into some of these types of schools, 
especially in urban settings (e.g., Abdulkadiroglu et al., 2016; and Eyles and Machin, forthcoming). 
If autonomous schools do deliver performance improvements, one key research question is whether 
they do so by improving the education levels of children enrolled in the school, or whether their 
extra autonomy means that they are better able to manipulate measures of school performance. In 
the English setting we study, one way to do this is to permanently exclude poorly performing 
students who bring down the school average. 

This question is directly studied in this paper, which contributes to the growing literature 
on possible test score manipulation by presenting results on a novel possible means of results 
manipulation to the existing work. It presents causal estimates of academy status on pupil 
exclusions and their strategic nature, and asks whether it is indeed possible that the achievement 
gains seen in some academy schools can be attributed to this form of strategic exclusion. Whilst it 
is the case that pre-2010 academies did exclude more pupils enrolled in the final year of secondary 
school, it is not possible to uncover evidence that this differential exclusion facilitated manipulation 


of overall school performance. 
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The analysis of the post-2010 academies also reveals that these schools started to exclude 
more following conversion. However, the estimated effect here is much smaller than for the pre- 
2010 academies, and this is most likely due to the very different nature of the academies programme 
after 2010. In fact, post-2010 academies were much better schools prior to the academy conversion, 
thus it is plausible that they did not face the kind of behavioural problems and pressure for 
improvement faced by the pre-2010 academies. 

In conclusion, as some popular (albeit non-evidence based) public debate has sometimes 
argued, academy schools in England do indeed show higher rates of pupil permanent exclusion 
than otherwise comparable non-academy schools. However, the focus on the initial batch of 
academy schools in England, which were introduced as a remedial school improvement programme 
in the 2000s and that recorded steep performance improvements following conversion, reveals that 
exclusion was not a means of facilitating better outcomes for autonomous schools in the published 
league tables. Rather it seems to be part and parcel of the disciplinary behaviour procedures that 
these schools implement (i.e., an integral part of the “No Excuses” culture some academies 
operate). This finding is corroborated by the fact that we see much smaller gaps in permanent 
exclusion between the second (post-2010) conversions to academies, which were not 
disadvantaged schools with the behaviour problems that characterised the initial pre-2010 phase of 


academisation of the English secondary school sector. 
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Figure 1. IV Estimates of Impact on Permanent Exclusion in Year 11 in Pre-2010 Academies 


Permanent Exclusion in Pre-2010 Academies 
Any Permanent Exclusion of Pupils Enrolled in Year 11 
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Notes: Figure 1 shows event-study IV estimates of the causal effect of academy enrolment on 
year 11 permanent exclusion for Pre-2010 academies. c indicates year of academy conversion 
and 95 percent confidence intervals are reported together with point estimates. Robust standard 
errors were clustered at the school level. Control variables included are dummies for whether the 
pupils are eligible for free school meals, whether they are male, native English speakers, the 
pupils’ ethnicity group and a proxy for pre-determined school-level behaviour in the ITT school 
where the pupils pre-enrolled, entered together with KS2 test scores and a dummy variable for 
pupils for whom KS2 data is unavailable. 
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Figure 2. IV Estimates of Impact on Permanent Exclusion in Year 11 in Post-2010 Academies 
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Notes: Figure 2 shows event-study IV estimates of the causal effect of academy enrolment on 
year 11 permanent exclusion for Post-2010 academies. c indicates year of academy conversion 
and 95 percent confidence intervals are reported together with point estimates. Robust standard 
errors were clustered at the school level. Control variables included are dummies for whether the 
pupils are eligible for free school meals, whether they are male, native English speakers, the 
pupils’ ethnicity group and a proxy for pre-determined school-level behaviour in the ITT school 
where the pupils pre-enrolled, entered together with KS2 test scores and a dummy variable for 
pupils for whom KS2 data is unavailable. 
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Figure 3. School-Specific IV Estimates of Impact on Key-Stage 4 Test Scores against School-Specific IV Estimates of Impact 
on Strategic Permanent Exclusion in Year 11 in Pre-2010 Academies 
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Notes: Figure 3 shows school-specific IV estimates of the causal effect of academy enrolment on 
year 11 strategic permanent exclusion plotted against school-specific IV estimates of the causal 
effect of academy enrolment on Key-Stage 4 test score for Pre-2010 academies. A linear function 
fitting the distribution of school-specific IV estimates is also shown in Figure 3. For each Pre-2010 
academy in the analysis, the horizontal axis shows the IV estimate of the causal effect of academy 
enrolment on year 11 strategic permanent exclusion, whereas the vertical axis shows school- 
specific IV estimates of the causal effect of academy enrolment on Key-Stage 4 test score. 
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Table 1. Pupils Permanently Excluded From Secondary School, 
School Years 2001/02 to 2014/15 


Year 7 to 11 Year 7 Year 8 Year 9 Year 10 Year 11 


A. Any Permanent Exclusion 


Number 79868 6356 14835 23042 25898 9737 
Share of Year 7 to 11 1.00 0.08 0.19 0.29 0.32 0.12 
Percent Rate 0.197 0.079 0.184 0.283 0.315 0.121 
B. Pre-January Census Permanent 

Exclusion 

Number 31853 999 4834 8124 10072 7554 
Share of Year 7 to 11 1.00 0.03 0.15 0.26 0.32 0.24 
Percent Rate 0.079 0.012 0.060 0.100 0.122 0.094 


Notes: Table | shows the distribution of permanent exclusions across pupils in state-maintained secondary schools in England 
from 2001/02 to 2014/15. The total number of pupils that were permanently excluded in year 11 from 2001/02 to 2014/15 
was calculated using the NPD data. Table | reports both the number of pupils that were permanently excluded at any time in 
the school year and the number of pupils that were excluded in the time window from the beginning of the school year to the 
January Census. The NPD data was also used to calculate the total number of pupils enrolled in state-maintained schools in 
England both at the aggregate and separately for each school year. The share of year 7 to 11 in each year shows the share of 
pupils excluded in a given school year out of the total number of permanently excluded pupils in the school years 7 to 11. For 
each school year, the Percent Rate shows the percent rate of pupils permanently excluded out of the total of the pupils enrolled 
in a given school year. 
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Table 2. Descriptive Determinants of Discipline Outcomes in Year 11 


Any precey T Fraction of Persistent 
Permanent see emporary Unauthorised Unauthorised 
; Permanent Exclusion 
Exclusion ; Absence Absence 
Exclusion 
@) (2) (3) (4) (5) 
FSM Eligible 2.133*** 2.235*** 1.853*** 1.966*** 3.233*** 
(0.053) (0.062) (0.009) (0.005) (0.021) 
Male 3.769% ** 3.593 *** 2.500*** 0.987*** 0.896*** 
(0.099) (0.105) (0.011) (0.002) (0.005) 
English Native 2.018*** 2.193 *** 1.863*** 1.088*** 2.347*** 
Speaker (0.084) (0.104) (0.016) (0.004) (0.032) 
KS2 Test Score 0.651*** 0.639*** 0.636*** 0.690*** 0.570*** 
(0.006) (0.007) (0.001) (0.001) (0.002) 
Year Fixed Effects Yes Yes Yes Yes Yes 
Number of Pupils 7336508 7336508 5219629 5219629 5219629 
Number of Schools 3008 3008 2949 2949 2949 


Notes: Table 2 reports estimates of the determinants of behavioural outcomes for year 11 pupils. 
Odds ratios are reported in all columns. Robust standard errors are reported in parentheses. * indicates 
significance at 10 percent, ** indicates significance at 5 percent, *** indicates significance at | 
percent. Control variables included are dummies for whether the pupils are eligible for free school 
meals, whether they are male, native English speakers, the pupils’ ethnicity group, entered together 
with KS2 test scores and a dummy variable for pupils for whom KS2 data is unavailable. 


50 


Table 3. Descriptive Determinants of Permanent Exclusion in Year 11 by School Type 


All Pre-2010 Post-2010 — Post-2010 Post-2010 Future To 
Schools Academies Academies Sponsored Converter Be 
Academies Academies Academies 
(1) (2) (3) (4) (5) (6) 
A. Any Permanent Exclusion 
FSM Eligible 2.133*** 1.671*** 2.365*** 1.720*** 2.464*** 2.118*** 
(0.053) (0.166) (0.201) (0.330) (0.233) (0.057) 
Male 3.769*** 3:215*** 3.967*** 4.264*** 3.904*** 3.780*** 
(0.099) (0.358) (0.354) (0.981) (0.378) (0.107) 
Native English Speaker 2.018*** 2.161*** L717 t* 1.698 1.803*** 2.048*** 
(0.084) (0.345) (0.244) (0.553) (0.287) (0.093) 
KS2 Test Score 0.651*** 0.780*** 0.592*** 0.720*** 0.580*** 0.654*** 
(0.006) (0.037) (0.019) (0.062) (0.021) (0.007) 
B. Pre-January Census 
Permanent Exclusion 
FSM Eligible DIZS*** 1.743 *** 2.495*** 1.633** 2.657*** ORO hits 
(0.062) (0.188) (0.229) (0.340) (0.271) (0.068) 
Male 3.593*** 3.556*** 3.805*** 4.212*** 3.725*** 3.565*** 
(0.105) (0.444) (0.366) (1.041) (0.389) (0.113) 
Native English Speaker 2.193*** 2.501 *** 1.630*** 1.656 1.703*** 2252 *** 
(0.104) (0.436) (0.249) (0.589) (0.291) (0.118) 
KS2 Test Score 0.639*** 0.794*** 0.577*** 0.700*** 0.566*** 0.641 *** 
(0.007) (0.042) (0.020) (0.065) (0.022) (0.008) 
Year Fixed Effects Yes Yes Yes Yes Yes Yes 
Number of Pupils 7336508 229453 839466 76256 763210 6267589 
Number of Schools 3008 208 1375 205 1170 3008 


Notes: Table 3 reports estimates of the determinants of permanent exclusion in year 11 by school type. In Panel A, the 
dependent variable is permanent exclusion at any time during year 11. In Panel B, the dependent variable is permanent 
exclusion before the January Census during year 11. Odds ratios are reported in all columns. Robust standard errors are 
reported in parentheses. * indicates significance at 10 percent, ** indicates significance at 5 percent, *** indicates 
significance at 1 percent. Control variables included are dummies for whether the pupils are eligible for free school meals, 
whether they are male, native English speakers, the pupils’ ethnicity group, entered together with KS2 test scores and a 
dummy variable for pupils for whom KS2 data is unavailable. 
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Table 4. Academy Conversions By School Year 


All Schools With Full Data (Pre- and Post-Academy Conversion) 


All 2002/03 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 ~—2009/10 2010/11 


A. Pre-2010 Treatment Academies, 94 3 6 2, 7 14 25 37 0 0 
Academies Convert in Labour Years 
Control To Be Academies, 114 0 0 0 0 0 0 0 58 56 
Convert in 2009/10 to 2010/11 
All 2010/11 2011/12 2012/13 2013/14 = 2014/15 2015/16 =—2016/17 
B. Post-2010 Treatment Academies, 157 0 30 58 69 0 0 0 
Sponsored Convert in 2010/11 to 2013/14 
Academies 
Control To Be Academies, 97 0 0 0 0 48 44 5 
Convert in 2014/15 to 2016/17 
All 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 
C. Post-2010 Treatment Academies, 1116 64 640 303 109 0 0 0 
Converter Convert in 2010/11 to 2013/14 
Academies 
Control To Be Academies, 104 0 0 0 0 54 42 8 


Convert in 2014/15 to 2016/17 


Notes: Table 4 shows the timeline of academy conversion of the secondary schools in England that were included in our analysis. The timeline is presented 
separately for pre-2010 academies, post-2010 sponsored academies and post-2010 converter academies. 


Source is https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/175360/academies_annual_report_2010-11.pdf. 
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Table 5. Difference-in-Differences Before and After Academy Conversion, 
Any Permanent Exclusion in Year 11 


Any Permanent Exclusion in Year 11 (X 100) 


Academies Future To Academies - Unconditional 
Be Future To Be DiD 
Academies Academies Estimate 
(1) (2) (3)=()-@) (4) 

A. Pre-2010 Academies 
Pre-Period 0.184 0.208 -0.024 DiD = 
Post-Period 0.348 0.208 0.140 0.164** 
Post - Pre 0.164 0.000 (0.064) 
Number of Schools 94 114 208 
B. Post-2010 Academies 
Pre-Period 0.092 0.082 0.010 DiD = 
Post-Period 0.087 0.070 0.017 0.007 
Post - Pre -0.005 -0.012 (0.012) 
Number of Schools 1273 201 1474 
C. Post-2010 Sponsored Academies 
Pre-Period 0.137 0.113 0.024 DiD = 
Post-Period 0.144 0.105 0.039 0.015 
Post - Pre 0.007 -0.008 (0.036) 
Number of Schools 157 97 254 
D. Post-2010 Converter Academies 
Pre-Period 0.086 0.062 0.024 DiD = 
Post-Period 0.082 0.050 0.032 0.008 
Post - Pre -0.004 -0.012 (0.011) 
Number of Schools 1116 104 1220 


Notes: Table 5 shows the results of an unconditional difference-in-differences exercise comparing year 
11 permanent exclusion three years before and after academy conversion in treatment group schools and 
control group schools. Panel A shows the result of this exercise for Pre-2010 academies. Panel B shows 
the result of this exercise for Post-2010 academies. Panel C shows the result of this exercise for Post- 
2010 sponsored academies and Panel D shows the result of this exercise for Post-2010 converter 
academies. Robust standard errors clustered at the school level are reported in parentheses. * indicates 
significance at 10 percent, ** indicates significance at 5 percent, *** indicates significance at | percent. 
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Table 6. Fraction of Pupils Sitting Exams in Excluding Schools 


Academies and Non-Academies 


Academies and Future To Be Academies 


(1) (2) (3) (4) 
Number of Year Share Sitting Number of Year 11 Share Sitting 
11 Permanent Exam in Permanent Exam in 
Exclusions Excluding School Exclusions Excluding School 

A. Pre-2010 Academies 

All 6276 0.245 713 0.231 
Pre-January Census 4771 0.110 536 0.084 
Post-January Census 1505 0.671 177 0.678 

B. Post-2010 Academies 

All 5564 0.219 2524 0.227 
Pre-January Census 4442 0.102 2013 0.105 
Post-January Census 1122 0.682 511 0.710 

C. Post-2010 Sponsored Academies 

All 5274 0.220 591 0.208 
Pre-January Census 4189 0.100 470 0.106 
Post-January Census 1085 0.682 121 0.603 

D. Post-2010 Converter Academies 

All 5363 0.218 1933 0.233 
Pre-January Census 4289 0.102 1543 0.104 
Post-January Census 1074 0.684 390 0.741 


Notes: Table 6 shows the fraction of permanently excluded pupils that sit exams in the excluding school by period of permanent 
exclusion in year 11. The total number of pupils that were permanently excluded in year 11 from 2001/2 to 2014/15 was 
calculated using the NPD data. Table 6 reports the number of pupils that were permanently excluded at any time in the school 
year, the number of pupils that were excluded in the time window from the beginning of the school year to the January Census, 
and the number of pupils that were excluded after the January Census. Since the NPD data reports the exact school premises 
where the GCSE exams were sat by each pupil, the NPD data was also used to calculate the share of pupils that were 
permanently excluded and that sat the GCSE exams in the excluding school. 
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Table 7. Difference-in-Differences Before and After Academy Conversion, 
Strategic Permanent Exclusion in Year 11 


Strategic Permanent Exclusion in Year 11 (X 100) 


Academies Future To Academies - Future Unconditional 


Be To Be Academies DiD 
Academies Estimate 
(1) (2) (3)=()-@) (4) 

A. Pre-2010 Academies 
Pre-Period 0.157 0.154 0.003 DiD = 
Post-Period 0.272 0.167 0.105 0.102* 
Post - Pre 0.115 0.013 (0.059) 
Number of Schools 94 114 208 
B. Post-2010 Academies 
Pre-Period 0.073 0.069 0.004 DiD = 
Post-Period 0.071 0.054 0.017 0.013 
Post - Pre -0.002 -0.015 (0.010) 
Number of Schools 1273 201 1474 
C. Post-2010 Sponsored Academies 
Pre-Period 0.113 0.091 0.022 DiD = 
Post-Period 0.118 0.083 0.035 0.013 
Post - Pre 0.005 -0.008 (0.031) 
Number of Schools 157 97 254 
D. Post-2010 Converter Academies 
Pre-Period 0.068 0.055 0.013 DiD = 
Post-Period 0.067 0.037 0.030 0.017* 
Post - Pre -0.001 -0.018 (0.009) 
Number of Schools 1116 104 1220 


Notes: Table 7 shows the results of an unconditional difference-in-differences exercise comparing year 11 
strategic permanent exclusion three years before and after academy conversion in treatment group schools and 
control group schools. Panel A shows the result of this exercise for Pre-2010 academies. Panel B shows the 
result of this exercise for Post-2010 academies. Panel C shows the result of this exercise for Post-2010 
sponsored academies and Panel D shows the result of this exercise for Post-2010 converter academies. Robust 
standard errors clustered at the school level are reported in parentheses. * indicates significance at 10 percent, 
** indicates significance at 5 percent, *** indicates significance at 1 percent. 
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Table 8. Estimates of Academy Impact on Permanent Exclusion in Year 11 


Any Permanent Exclusion 
in Year 11 (X 100) 


Strategic Permanent Exclusion 
in Year 11 (X 100) 


Academies vs 
Control Schools 


OLS ITT IV OLS ITT IV 
(1) (2) (3) (4) (5) (6) 
Academy 0.053*** 0.088*** 0.091*** 0.051*** 0.080*** 0.083*** 
(0.017) (0.017) (0.018) (0.015) (0.016) (0.016) 
Control Variables Yes Yes Yes Yes Yes Yes 
School Fixed Effects Yes Yes Yes Yes Yes Yes 
Year Fixed Effects Yes Yes Yes Yes Yes Yes 
Number of Observations 4677340 4677340 4677340 4677340 4677340 4677340 
Number of Schools 1682 1682 1682 1682 1682 1682 
First Stage Coefficient on ITT 0.965*** 0.965*** 
(0.001) (0.001) 


Notes: Table 8 shows estimates of the causal effect of academy enrolment on year 11 permanent exclusion and on year 
11 strategic permanent exclusion for all academies. Robust standard errors (clustered at the school level) are reported 
in parentheses. Control variables included are dummies for whether the pupils are eligible for free school meals, 
whether they are male, native English speakers, the pupils’ ethnicity group and a proxy for pre-determined school-level 
behaviour in the ITT school where the pupils pre-enrolled, entered together with KS2 test scores and a dummy variable 
for pupils for whom KS2 data is unavailable. * indicates significance at 10 percent, ** indicates significance at 5 


ercent, *** indicates significance at 1 percent. 
° 
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Table 9. Estimates of Academy Impact on Any Permanent Exclusion in Year 11 


Any Permanent Exclusion in Year 11 (X 100) 


Academies vs 
Control Schools 


Pre-2010 Academies 


Post-2010 Academies 


Post-2010 Sponsored Academies 


Post-2010 Converter Academies 


OLS ITT IV OLS ITT IV OLS ITT IV OLS ITT IV 
() (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) 
Academy 0.303*** 0.319*** 0.343*** 0.015 0.051***  0.052*** 0.038 0.059 0.062 0.011 0.050***  0.051*** 
(0.084) (0.084) (0.090) (0.014) (0.014) (0.014) (0.038) (0.038) (0.040) (0.014) (0.014) (0.014) 
Control Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Variables 
School Fixed Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Effects 
Year Fixed Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Effects 
Number of 1304990 1304990 1304990 3372350 3372350 83372350 746328 746328 746328 2625995 2625995 2625995 
Observations 
Number of 208 208 208 1474 1474 1474 254 254 254 1220 1220 1220 
Schools 
First Stage 0.929*** 0.971 *** 0.946*** 0.978 *** 
Coefficient on (0.004) (0.001) (0.246) (0.065) 


ITT 


Notes: Table 9 shows estimates of the causal effect of academy enrolment on year 11 permanent exclusion separately by group of academies. Robust standard errors 
(clustered at the school level) are reported in parentheses. Control variables included are dummies for whether the pupils are eligible for free school meals, whether they 
are male, native English speakers, the pupils’ ethnicity group and a proxy for pre-determined school-level behaviour in the ITT school where the pupils pre-enrolled, 
entered together with KS2 test scores and a dummy variable for pupils for whom KS2 data is unavailable. * indicates significance at 10 percent, ** indicates significance 
at 5 percent, *** indicates significance at 1 percent. 
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Table 10. Estimates of Academy Impact on Strategic Permanent Exclusion in Year 11 


Strategic Permanent Exclusion in Year 11 (X 100) 


Pre-2010 Academies Post-2010 Academies Post-2010 Sponsored Academies Post-2010 Converter Academies 
OLS ITT IV OLS ITT IV OLS ITT IV OLS ITT IV 
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) 
Academy 0.260*** 0.262*** 0.282*** 0.020* 0.051*** 0.052*** 0.034 0.051 0.054 0.018 0.052***  0.053*** 
(0.081) (0.081) (0.087) (0.012) (0.012) (0.012) (0.035) (0.035) (0.037) (0.012) (0.012) (0.012) 
Control Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Variables 
School Fixed Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Effects 
Year Fixed Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Effects 
Number of 1304990 ~=—6.1304990-—s: 1304990) )=—s- 33372350) )=—- 3372350) »3=—- 3372350 »3=—& 746328 )3=—_ 746328 746328 2625995 2625995 2625995 
Observations 
Number of 208 208 208 1474 1474 1474 254 254 254 1220 1220 1220 
Schools 
First Stage 0.929*** 0.971*** 0.946*** 0.978 *** 
Coefficient on (0.004) (0.001) (0.246) (0.065) 
ITT 


Notes: Table 10 shows estimates of the causal effect of academy enrolment on year 11 strategic permanent exclusion separately by group of academies. Robust 
standard errors (clustered at the school level) are reported in parentheses. Control variables included are dummies for whether the pupils are eligible for free 
school meals, whether they are male, native English speakers, the pupils’ ethnicity group and a proxy for pre-determined school-level behaviour in the ITT 
school where the pupils pre-enrolled, entered together with KS2 test scores and a dummy variable for pupils for whom KS2 data is unavailable. * indicates 
significance at 10 percent, ** indicates significance at 5 percent, *** indicates significance at 1 percent. 
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Table 11. Estimates of Academy Impact on Key Stage 4 Test Scores 


Key Stage 4 Standardised Test Scores, End of Secondary School 


Pre-2010 Academies Post-2010 Academies 
OLS ITT IV OLS ITT IV 
(1) (2) (3) (4) (5) (6) 
Academy 0.145*** 0.097*** 0.104*** -0.008 -0.004 -0.004 
(0.028) (0.028) (0.030) (0.013) (0.013) (0.014) 
Control Variables Yes Yes Yes Yes Yes Yes 
School Fixed Effects Yes Yes Yes Yes Yes Yes 
Year Fixed Effects Yes Yes Yes Yes Yes Yes 
Number of Observations 1304990 1304990 1304990 3372350 3372350 3372350 
Number of Schools 208 208 208 1474 1474 1474 
First Stage Coefficient on ITT 0.927*** 0.970*** 
(0.004) (0.001) 


Notes: Table 11 shows estimates of the causal effect of academy enrolment on standardised Key-Stage 4 test score 
separately by group of academies. Robust standard errors (clustered at the school level) are reported in parentheses. 
Control variables included are dummies for whether the pupils are eligible for free school meals, whether they are 
male, native English speakers, the pupils’ ethnicity group, entered together with KS2 test scores and a dummy variable 
for pupils for whom KS2 data is unavailable. * indicates significance at 10 percent, ** indicates significance at 5 
percent, *** indicates significance at | percent. 
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Table 12. Simulation Estimates of Academy Impact on Key Stage 4 Test Scores for Pre-2010 Academies 


Pre-2010 Academy Impact on Key Stage 4 Standardised Test Scores 


Only 
Reinstating 
Pre-2010 Academies Original Excluded (2) + (3) + (3) + (3) + (3) + (3) + (3) + (3) + (3) + (3) + 
IV Pupils with Remove Top 5% 10% 20% 25% 30% 35% 40% 45% 50% 
Worst KS4 Negative Negative Negative Negative Negative Negative Negative Negative Negative 
Possible Performers Peer Peer Peer Peer Peer Peer Peer Peer Peer 
KS4 Score Effect Effect Effect Effect Effect Effect Effect Effect Effect 
() (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) 
Academy 0.104*** 0.103*** 0.102*** 0.097*** 0.091***  0.080** 0.074** 0.069* 0.063 0.058 0.052 0.046 
(0.030) (0.030) (0.030) (0.030) (0.031) (0.034) (0.036) (0.039) (0.042) (0.044) (0.048) (0.051) 
Control Variables Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
School Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 
Year Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 


Notes: Table 12 shows simulation IV estimates of the causal effect of academy enrolment on standardised Key Stage 4 test score for Pre-2010 academies. Column (1) reports the result 
in column (3) of Table 11 for purpose of comparison. Column (2) shows the IV estimate of the academy impact on KS4 outcome when strategically excluded pupils are reinstated in the 
excluding school with the worst possible KS4 outcome. Column (3) shows the IV estimate of the academy impact on KS4 outcome when, in addition to the exercise in column (2), for 
a given number of excluded pupils in school s in year t, an equivalent number of Top KS4 performers is removed from the analysis. Columns (4) to (12) show the IV estimates of the 
academy impact on KS4 outcome when, in addition to the exercise in column (3), an increasingly negative peer effect of each excluded pupil on all other pupils in school s and year t is 
assumed. Robust standard errors (clustered at the school level) are reported in parentheses. Control variables included are dummies for whether the pupils are eligible for free school 
meals, whether they are male, native English speakers, the pupils’ ethnicity group, entered together with KS2 test scores and a dummy variable for pupils for whom KS2 data is 
unavailable. * indicates significance at 10 percent, ** indicates significance at 5 percent, *** indicates significance at | percent. 
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Table 13. Further Empirical Tests for Pre-2010 Academies 


Strategic Permanent 
Exclusion in Year 11 


Key Stage 2 Standardised 
Test Scores, End of 
Primary School, Strategic 


Non-Strategic 
Permanent Exclusion 
in Year 11 (X 100) 


Cel) Permanent Excluded 
Pre-2010 Academies 
IV OLS IV 
(1) (2) (3) 
Academy 0.283*** 0.003 0.061** 
(0.086) (0.258) (0.030) 
Academy x Key-Stage 2 0.016 
(0.034) 
Control Variables Yes Yes Yes 
School Fixed Effects Yes Yes Yes 
Year Fixed Effects Yes Yes Yes 
Number of Observations 1304990 2023 1304990 
Number of Schools 208 156 208 
First Stage Coefficient on ITT 0.929*** 0.929*** 
(0.004) (0.004) 
First Stage Coefficient on ITT x -0.008*** 
Key Stage 2 (0.002) 


Notes: Column (1) of Table 13 shows IV estimates of the causal effect of academy enrolment interacted with prior Key- 
Stage 2 test score on year 11 strategic permanent exclusion for pre-2010 academies. Column (2) of Table 13 shows 
OLS estimates of the causal effect of academy enrolment on prior Key-Stage 2 test score for pupils that were 
strategically excluded in year 11 in pre-2010 academies. Column (3) of Table 13 shows estimates of the causal effect 
of academy enrolment on year 11 non-strategic permanent exclusion for pre-2010 academies. Robust standard errors 
(clustered at the school level) are reported in parentheses. In Columns (1) and (3), control variables included are 
dummies for whether the pupils are eligible for free school meals, whether they are male, native English speakers, the 
pupils’ ethnicity group and a proxy for pre-determined school-level behaviour in the ITT school where the pupils pre- 
enrolled, entered together with KS2 test scores and a dummy variable for pupils for whom KS2 data is unavailable. In 
Column (2), control variables included are dummies for whether the pupils are eligible for free school meals, whether 
they are male, native English speakers and the pupils’ ethnicity group. * indicates significance at 10 percent, ** indicates 
significance at 5 percent, *** indicates significance at | percent. 
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Appendix 
Data Description 
1. Data on Academy Schools 


For the analysis of the pre-2010 academies, we detected all schools that acquired academy status over 
the school years 2002/03 to 2010/11. For the analysis of the post-2010 academies, we detected all 
schools that acquired academy status over the school years 2010/11 to 2016/17. In order to do this, we 
used Department for Education (DfE) extracts that contain information on all academies that have 
converted or that are undergoing the process of academy conversion. The DfE extracts contain 
information on the opening date of the academy, its URN (a unique identifier for the school that enables 
us to identify it in several governmental data sources such as the National Pupil Database and the Pupil 
Level Annual Census data), the URN number of the predecessor school, the LAEstab number of the 
academy (an additional unique identifier that combines local authority specific and school specific 
numbers) and the LAEstab number of the predecessor school. 


For the analysis of pre-2010 academies, we used performance tables data from the Department for 
Education (DfE) to match in predecessor school types. In order to have a balanced panel of schools 
with KS4 results since 2002, we focused on academies that have some kind of predecessor school open 
since at least 1997 on. We identified 244 schools that became academies from the first 3 academy 
openings in 2002/03 to those that became academies by September 2010 (the beginning of the academic 
school year). For the analysis of Post-2010 academies, we used the information available online from 
the DfE’s Edubase to identify post-2010 academies that had a predecessor open since at least 2002 (this 
is a Department For Education database of all open and closed maintained schools in England; see 
http://www.education.gov.uk/edubase/home.xhtml). For the analysis of both pre- and post-2010 
academies, in order for our sample to be balanced for intake, we included in our analysis only 
academies that enrol pupils in year 7, 1.e., at the beginning of secondary school. In all cases, we did not 
include in our analysis brand new academy schools as they have no predecessor school. Likewise, we 
did not include in our analysis previously independent schools, since pupils in these schools do not 
have exam information at KS4. 


For the analysis of pre-2010 academies, the final sample contains 106 (of which we use the 94 who 
were not CTCs in their predecessor state) treatment schools (those that opened as academies prior to, 
or in, September 2008) and 114 control schools with observations ranging over the years 2001/02- 
2008/09. None of our control schools become academies during these sample years, but all converted 
by September 2010. For the analysis of post-2010 academies, the final sample contains 254 post-2010 
sponsored academies and 1220 post-2010 converter academies. For the study of post-2010 sponsored 
academies, we define a sample of 157 treatment schools that opened by December 2013 (i.e., by the 
academic year 2013/14), and a group of 97 control schools that opened after December 2013 but were 
either open or in the pipeline for opening by the spring 2016. Similarly for the study of post-2010 
converter academies, we define a sample of 1116 treatment schools that opened by December 2013 
(1.e., by the academic year 2013/14), and a group of 104 control schools that opened after December 
2013 but were either open or in the pipeline for opening by the spring 2016. 
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2. Pupil Level Data 


We use data from PLASC (pupil level annual schools census) and the NPD (national pupil database) 
from the school years 2001/2 to 2013/14. The NPD reports data on all key stage 2 (KS2) and key stage 
4 (KS4) exams sat at the end of primary and secondary school respectively. We could identify each 
pupil across datasets using a unique reference number. The data also contains a unique LAEstab code 
of the school in which the pupil sat the exam: this is a key piece of information that we use to identify 
pupils that were excluded in year 11 and that ended up sitting exams in a different school. While the 
NPD contains data on pupils in examination years, PLASC contains a record for every pupil for each 
year that they are in the maintained school sector. PLASC data gives the pupil, year group and school 
as well as demographic variables such as ethnicity, gender, free school meal eligibility and special 
educational needs status. We can track pupils through secondary school using the unique pupil 
identifier. This identifier is common to the NPD enabling us to merge NPD and PLASC data. This 
gives a panel of pupils with their demographic information, their KS2 and KS4 test results and the 
school(s) that they attended from year 7 (first year of compulsory secondary education) through to year 
11 (final year of compulsory education). For the analysis of pre-2010 academies, we identified the 
pupils who attended the 208 treatment and control pre-2010 academies at some point from the school 
years 2001/2 to 2008/9, and we used information on the schools pupils attended in every secondary 
compulsory year of schooling”*, their demographic information and their exams results at KS4 and 
KS2. Our analysis focuses on those who are ITT in one of our 94 treatment schools or sit exams in one 
of our 208 control schools from the school years 2001/2 to 2008/9. For the analysis of post-2010 
sponsored academies, we identified the pupils who attended the 254 treatment and control post-2010 
sponsored academies from the school years 2005/6 to 2013/4. For the analysis of the post-2010 
converter academies, we identified the pupils who attended the 1220 treatment and control post-2010 
converter academies from the school years 2005/6 to 2013/4. For the analysis of post-2010 sponsored 
academies, our analysis focuses on those who are ITT or sit exams in one of our 254 sample schools 
from the school years 2005/6 to 2013/14. Similarly, for the analysis of post-2010 converter academies, 
our analysis focuses on those who are ITT or sit exams in one of our 1220 sample schools from the 
school years 2005/6 to 2013/14. We adopt the same empirical approach for the analysis of pre-2010 
and post-2010 academies respectively. 


3. Notes on Clustering 


For both pre-2010 and post-2010 academies, for each of our treatment and control schools we assign a 
unique number. It is possible that two pupils from different schools are given the same number should 
the two differing schools later become the same academy. We identify when schools merge by looking 
at linked schools in DfE’s Edubase. For pre-2010 academies, in one case a single school becomes two 
separate academies (North Westminster Community School splits into Paddington Academy and 
Westminster Academy in 2006). Pupils attending the predecessor school are randomly assigned one of 
the two numbers given to the two academies that open later. In all the analysis, pupils who leave the 
sample but are ITT or receive exposure are given a unique number equal to the school where they reach 
year 11. In estimated specifications, standard errors are clustered on this unique number. This results 


28 Strictly speaking this is not true. Some pupils enter the schooling system either from another country or from independent 
schools. We observe when the pupils enter but not precisely where they came from. These pupils are retained in our analysis. 
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in 208 clusters for the analysis of pre-2010 academies, in 254 clusters for the analysis of post-2010 
sponsored academies, and in 1220 clusters for the analysis of post-2010 converter academies. 


4. Outcome Variables and Attainment Measures 


The main outcome variable is permanent exclusion in year 11. The NPD contains all records of 
permanent exclusions from state-maintained schools in England. To be precise, pupil-level information 
on permanent exclusion is available since the academic year 2001/2, thus covering the entire period of 
our analysis. Information on temporary and lunchtime exclusion is available from the academic year 
2005/6, thus covering the entire period of our analysis for post-2010 academies, but not for pre-2010 
academies (i.e., we are unable to assess the impact of academy enrolment for the earliest 18 academies 
in our sample, that opened from 2002/3 to 2005/6). 


For each exclusion, the NPD reports the date of commencement of the exclusion, the reason for the 
exclusion, and the URN of the school from which the pupil was excluded. Data on temporary exclusion 
also includes information on the duration of the temporary exclusion. Although the data on exclusion 
do not report the school year of the excluded pupil, they contain the unique pupil identifier common to 
all NPD and PLASC datasets. We used this to combine pupil-level information on exclusion with 
information from PLASC and other NPD datasets. Although we used PLASC data collected in the 
January census, we were able to identify the schools that excluded pupils prior to the January census 
by comparing the information on the excluding school in the exclusion data with the school in which a 
pupil appeared in the January census. 


We also model the impact of academy enrolment on unauthorised absence from school. Our dependent 
variable in this case is calculated as the fraction of sessions missed by a pupil without authorization 
during the school year in year 11. We also model the impact of academy enrolment on the likelihood 
to be “persistently absent”. This is defined as a pupil that missed at least 20 percent of sessions during 
the school year in year 11. Even though the 2010 Coalition government changed the official definition 
of persistent truant in school, we keep this definition constant over the entire analysis of pre-2010 and 


post-2010 academies”’. 


We also look at the KS4 impact of academy enrolment. For this, we follow closely what Eyles and 
Machin (2015) do. The main KS4 qualification in the UK is the GCSE (General Certificate of 
Secondary Education). GCSEs are graded from A*-G. The current points score calculations give an A* 
a score of 58 and a Ga score of 16 with grades in between going up in increments of 6 between adjacent 
grades as follows: 


9 See, e.g., https://www.gov.uk/government/news/persistent-absence-government-changes-definition-to-deal-with- 
reality-of-pupil-absenteeism-in-schools 


64 


Grade Points Grade Points 


A* 58 D 34 

A 52 E 28 

B 46 F 22 

C 40 G 16 
New scale 


Prior to this an A* was given a score of 8 and G a score of | with scores rising in unit increments. 


Grade Points Grade Points 
A* 8 D 4 
A 7 E 3 
B 6 F 2 
C 5 G 1 
Old scale 


As well as GCSEs there are a wide range of equivalent qualifications focusing on more vocational 
subjects. These include GNVQs and BTecs. Depending upon the type of equivalent these are often 
worth multiple GCSEs and are often graded as a combination of GCSE grades i.e. a distinction in an 
intermediate GNVQ is equivalent to gaining two GCSEs with one at grade A and the other at grade 
A*2° The points score given to the qualification reflects the underlying GCSE grades that it is based 
upon so that under the new scoring system the aforementioned qualification would be given a score of 
110. The points system we use is as follows: 


Grade Points Grade Points 
A* 10 D 4 
A 8 E 3 
B 7 F 2 
C 6 G 1 


Scale used in the paper 


3° Most equivalents are graded as pass, merit or distinction but the Department for Education equates these categories, 
combinations of, A*-G grades. 
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The points system we use addresses some of the concerns expressed pertaining to the 16-58 and 1-8 
scales used over the course of our sample.*! The non-linearity reflects the fact that it appears hardest to 
jump from grades D to C and from A to A*. 


We cap points scores at best 8 qualifications. To do this we normalize raw point scores by their GCSE 
equivalent i.e. a qualification worth 4 GSCEs and 208 points (under the 16-58 scale) is normalized to 
be worth 52 points. We then convert these points to our new measure and rank them highest to lowest. 
We then add up the grade weightings (in terms of GCSEs), taking fractions of qualifications if need be, 
until we reach 8. All those in the top 8 are then multiplied through by their weight and summed to give 
the points score. 


Our decision to cap at 8 is motivated by two concerns. Total points scores have the problem that pupils 
can appear to do well by entering many exams and performing poorly in them. Similarly using, for 
instance, 5 best means that those who focus very narrowly on a small set of exams may appear better 
than those who perform well over a larger selection of subjects/qualifications. Our decision to cap at 8 
balances these two concerns. 


Finally, it is worth noting that our point measures create some notable discrepancies with the official 
method. For instance, an equivalent qualification worth two GCSEs graded CD is worth 74 points under 
the 16-58 scale meaning that it is worth more than a A* at GCSE. Using our system such a qualification 
is worth 10 points (the sum of the points scores for grades of C and D) — the equivalent of a GCSE at 
grade A*. A further example is a BTEC that is worth 76 points on the old scale and equivalent to 4 
GCSEs. This is the same as achieving grades of 2 Fs and 2 Gs. In our system this is equivalent to a 
point score of 6. Thus our points mean the qualification is the same as getting a C at GCSE whereas 
the old measure means that the qualification is again worth more than an A*. In general our system 
reduces the relative points scores of equivalent qualifications compared to the official method. Despite 
this our results remain unchanged when using the (standardized) old (1-8) and new (16-58) points 
systems and when using total rather than capped scores. 


31 We are grateful to Tim Leunig and Mike Treadaway for very helpful correspondence on this. 
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facademies-move-difficult-pupils-other-schools-boost-their-results 


Resources Community 


Academies move difficult pupils to other 
schools to boost their results, study 
suggests 


Richard Vaughan 


Share this 
13th November 2015 at 00:04 


GCSE Behaviour Accountability 


EXCLUSIVE 


Some secondaries are engineering moves of their most challenging students to neighbouring schools, in 
order to boost their own GCSE leaque table scores, new research suggests. 
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Academies accused of pushing difficult students out 


war 
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Viatch Liz MacKean’s full Newsnight report 
Academies, independent state schools whose growth is 
flourishing under the coalition, ore being accused of letting 


down their most vulnerable pupils. 


It is claimed that the pressure of performing well in the league 
tables is leading to some students being pushed out unofficially, 
without being permanently excluded, as Newsnight's Liz 
MacKean reports. 


By her own admission 15-year-old Chloe was a difficult pupil, telling 


me she got into trouble for “smoking, being on my phone”. 


Her mother Donna adds to the list 
remarks, trying to get everyone to laugh at her really 


“Disrupting classes, making silly 


Chioe was identified as having Special Educational Needs, but her 
behaviour led to her academy in Lincolnshire apparently washing its 


hands of her 


She was at home for an entire 
term before eventually being sent 
to an alternative provision with 
other pupils deemed not to fit intc 


mainstream schooling 
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League tables ‘mislead on school 
success’ 


By Marc Ashdown 
BSC London News 


© 31 January 2017 | London ¥ ww [~) | < Share 


THINKSTOCK 


League tables are currently worked out frorm how many children were on-roll in January of their final year 


Some schools would have performed much worse than official league tables 
show if “churn rates” had been taken into account, it has been claimed. 
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D.C. charter schools expel students at far 
higher rates than traditional public 
schools 


Ss D.C. school documents reveal charter system expulsion gap 


G+ =_ 


By Emma Brown 


The District’s public charter schools have expelled students at a far higher rate than 


the city’s traditional public schools in recent years, according to school data, 
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Additional Tables 


Fractions of Pupils that Sit Exams in Excluding Schools 


Table A1. Fraction of Pupils that Sit Exams in the Excluding School by Month of 
Permanent Exclusion 


Month of Post-2010 Academies 
Permanent | Pre-2010 Academies and All Schools Poste0ly Sponsored Post-2010 Convertes 
. Academies and All Academies and All 
Exclusion and All Schools 
; Schools Schools 
in Year 11 
Share Share Share Share 
who Sit Numberof | whoSit Numberof | whoSit Numberof | whoSit Number of 
Exams in Permanent | Examsin Permanent | Examsin Permanent | Examsin Permanent 
Excluding Exclusions | Excluding Exclusions | Excluding Exclusions | Excluding Exclusions 
School School School School 

September 0.030 1,079 0.045 1,016 0.046 949 0.044 979 
October 0.048 1,118 0.059 1,077 0.057 1,019 0.060 1,044 
November 0.106 1,363 0.099 1,328 0.094 1,262 0.099 1,277 
December 0.197 802 0.172 699 0.169 650 0.167 681 
January 0.421 801 0.395 582 0.391 562 0.395 552 
February 0.651 484 0.622 410 0.623 398 0.629 399 
March 0.788 392 0.795 308 0.8 295 0.796 289 
April 0.782 110 0.853 68 0.846 65 0.853 68 
May 0.873 63 0.947 57 0.945 55 0.945 55 
June 0.789 38 0.938 16 0.938 16 0.938 16 
July 0.615 26 0.333 3 0.333 3 0.333 3 
Total 0.245 6,276 0.219 5,564 0.220 5,274 0.218 5,363 


Notes: Table Al shows the fraction of pupils in our treatment schools and in all other schools that sit GCSE exams in 


the excluding school by month of permanent exclusion in year 11. 
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Table A2. Fraction of Pupils that Sit Exams in the Excluding School by Month of 
Permanent Exclusion 


Month of 
Permanent | Pre-2010 Academies | Post-2010 Academies Esso Sponsored Fostcont Converter 
. Academies and Academies and 
Exclusion and Control Schools and Control Schools 
. Control Schools Control Schools 
in Year 11 
a Share Share Share 
he Number of | whoSit | Numberof| whoSit Numberof| whoSit | Number of 
Sit Exams } : ; 
a Permanent | Examsin Permanent | Examsin Permanent | Examsin Permanent 
: Exclusions | Excluding Exclusions | Excluding Exclusions | Excluding Exclusions 
Excluding 
School School School 
School 
September 0.015 130 0.044 477 0.067 105 0.038 372 
October 0.030 133 0.066 487 0.073 123 0.063 364 
November 0.107 131 0.097 609 0.106 141 0.094 468 
December 0.173 104 0.177 284 0.152 66 0.183 218 
January 0.418 91 0.410 278 0.300 70 0.447 208 
February 0.667 63 0.686 188 0.667 45 0.692 143 
March 0.773 44 0.818 138 0.794 34 0.817 104 
April 0.75 8 0.875 32 0.667 3 0.897 29 
May 1 3 0.95 20 0 1 1 19 
June 0.75 4 0.9 10 1 2 0.875 8 
July 0.5 2 0 1 0 1 - 0 
Total 0.231 713 0.227 2524 0.208 591 0.234 1,933 


Notes: Table A2 shows the fraction of pupils in our treatment and control schools that sit GCSE exams in the excluding 
school by month of permanent exclusion in year 11. 
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Balancing Tests 


Table B1. Pre-Conversion Characteristics and Tests of Balancing for Pre-2010 Academies 


Percent Percent Percent Key stage 2 
eligible for Percent Percent incidence of incidence of points score Number of 
free school male white permanent strategic permanent (mean) schools 
meals exclusion (X 100) — exclusion (X 100) 
All Schools 0.137 0.506 0.829 0.070 0.037 63.457 3003 
All Pre-2010 Academies” 0.240 0.524 0.773 0.123 0.051 56.807 220 
Current academies 
(éeatment aroun) 0.263 0.536 0.730 0.118 0.031 55.408 94 
ne 0.232 0.515 0.812 0.135 0.070 56.480 114 
Difference 0.031 0.021 -0.082* -0.017 -0.039 -1.072 
(0.019) (0.015) (0.040) (0.056) (0.027) (0.798) 


Notes: Standard errors clustered at school level reported in parentheses. Both panels refer to characteristics in the 2001/02 school year. The top panel is maintained schools in the 
UK, which do not convert to academies prior to, or in, the academic year 2010/11. All variables refer to characteristics of the year 11 cohort in 2001/02 i.e. year 11 pupils in the 
school year 2001/02, before any academies had opened. * denotes significance at the 5% level, ** at the 1% level. 


>? This includes 12 conversions from City Technology Colleges (CTCs) that, similarly to Eyles and Machin (2015), we do not include in the regression analysis. 
This is because these schools were already very autonomous and they were performing well prior to conversion to academy status (Eyles and Machin 2015). 
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Table B2. Pre-Conversion Characteristics and Tests of Balancing for Post-2010 Academies 


Percent Percent Percent Key stage 2 
eligible for Percent Percent incidence of incidence of points score Number of 
free school male white permanent strategic permanent (mean) schools 
meals exclusion (X 100) — exclusion (X 100) 
All Schools 0.140 0.506 0.776 0.116 0.093 64.443 1371 
Post-2010 Academies (Pre-conversion) 0.107 0.505 0.817 0.097 0.071 66.368 1474 
Current academies 
0.102 0.503 0.816 0.098 0.071 66.716 1273 
(treatment group) 
BUTS eee: 0.141 0.516 0.820 0.088 0.070 64.196 201 
(control group) 
Difference -0.039** -0.013 -0.004 0.010 0.001 2.520** 
(0.009) (0.010) (0.019) (0.019) (0.018) (0.346) 


Notes: Standard errors clustered at school level reported in parentheses. Both panels refer to characteristics in the 2009/10 school year. The top panel is maintained schools in the 
UK, which do not convert to academies prior to, or in, the academic year 2016/17. All variables refer to characteristics of the year 11 cohort in 2009/10 i.e. year 11 pupils in the 
school year 2009/10, before any academies had opened. * denotes significance at the 5% level, ** at the 1% level. 
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Table B3. Pre-Conversion Characteristics and Tests of Balancing for Post-2010 Sponsored Academies 


Percent Percent Percent Key stage 2 
eligible for Percent Percent incidence of incidence of points score Number of 
free school male white permanent strategic permanent (mean) schools 
meals exclusion (X 100) — exclusion (X 100) 
All Schools 0.117 0.504 0.794 0.102 0.078 65.833 2591 
Pose DOL Sponsors Academics 0.180 0.522 0.835 0.146 0.114 61.568 254 
(Pre-conversion) 
Current academies 
0.181 0.520 0.840 0.152 0.124 60.886 157 
(treatment group) 
Eulture poademles 0.178 0.525 0.828 0.136 0.098 62.627 97 
(control group) 
Difference 0.003 -0.005 0.012 0.016 0.026 -1.742** 
(0.016) (0.013) (0.030) (0.046) (0.041) (0.451) 


Notes: Standard errors clustered at school level reported in parentheses. Both panels refer to characteristics in the 2009/10 school year. The top panel is maintained schools in the 
UK, which do not convert to academies prior to, or in, the academic year 2016/17. All variables refer to characteristics of the year 11 cohort in 2009/10 i.e. year 11 pupils in the 
school year 2009/10, before any academies had opened. * denotes significance at the 5% level, ** at the 1% level. 
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Table B4. Pre-Conversion Characteristics and Tests of Balancing for Post-2010 Converter Academies 


Percent Percent Percent Key stage 2 
eligible for Percent Percent incidence of incidence of points score Number of 
free school male white permanent strategic permanent (mean) schools 
meals exclusion (X 100) — exclusion (X 100) 
All Schools 0.147 0.508 0.786 0.120 0.096 63.994 1625 
Posie 201) Sener et neadomies Wiis: 0.092 0.501 0.813 0.087 0.062 67.320 1220 
conversion) 
Current academies 
0.091 0.501 0.812 0.091 0.064 67.488 1116 
(treatment group) 
Tali seduce 0.108 0.508 0.813 0.044 0.044 65.581 104 
(control group) 
Difference -0.017 -0.007 -0.001 0.047** 0.020 1.907** 
(0.010) (0.016) (0.025) (0.016) (0.016) (0.473) 


Notes: Standard errors clustered at school level reported in parentheses. Both panels refer to characteristics in the 2009/10 school year. The top panel is maintained schools in the 
UK, which do not convert to academies prior to, or in, the academic year 2016/17. All variables refer to characteristics of the year 11 cohort in 2009/10 i.e. year 11 pupils in the 
school year 2009/10, before any academies had opened. * denotes significance at the 5% level, ** at the 1% level. 
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